muppeteer

Muppeteer

Muppeteer is a visual regression testing framework for running UI tests in Chrome. It's composed of a number of modules:

• Mocha - a test runner framework
• Chai - an assertion library
• Puppeteer - a library for interacting with Chrome and web pages
• Puppeteer Extensions - an extension library for Puppeteer with convenience functions
• Pixelmatch - a pixel-level image comparison library

In addition, it provides the following core features:

• Visual Regression Testing - a screenshot-based image comparison module that hooks onto the assertion API. Read on for more discussion on this.
• Test Interface - a modification of Mocha's BDD interface with built-in browser setup steps and other user configurable hooks
• Test Launcher - a CLI and configuration function for launching test suites

Muppeteer's main goal is to abstract the, often, tedious boilerplate setup code needed to write tests with Puppeteer, and provide a convenient API for testing UI functionality. It was inspired by the PhantomCSS and CasperJS libraries.

This framework is in beta pre-release currently. It is subject to some breaking changes until the API is finalised. There is also little test coverage as of yet, but it is in the works. Use with caution.

• Configuration

  • CLI

  • Configuration function

• Puppeteer API

• Puppeteer Extensions

• Example test case

  • Passing test output

  • Failing test output

  • Understanding visual failures

Configuration

You can configure Muppeteer via the CLI or a configuration function

CLI

The CLI script can be referenced at lib/test-launcher-cli.

It is run like node <<path-to-muppeteer>>/lib/test-launcher-cli <<args>>

Example

 "scripts": {
    "test": "node node_modules/muppeteer/lib/test-launcher-cli --t tests --f test.js --r tests/report"
  }

See Options

Configuration function

The configuration can be referenced at src/test-launcher.

Example

const ConfigureLauncher = require('../src/test-launcher');
const path = require('path');
const testsPath = path.join(__dirname, 'tests');

ConfigureLauncher({
        testDir: testsPath,
        testFilter: 'test.js',
        reportDir: `${testsPath}/report`,
        visualThreshold: 0.05,
        useDocker: true,
        onFinish: () => {
            // do something after the tests have complete
        }
    }
).launch();

Options

Note: Only options with -- can be run with the proceeding flag in the CLI interface.

• testDir (--t): The directory for Mocha to look for the test files
• testFilter (--f): Allows you to pass in some text to filter the test file name, it's just a substring match, nothing fancy
• shouldRebaseVisuals (--b): A flag to tell the visual regression engine to replace the existing baseline visuals
• reportDir (--r): The directory for the Mocha reporter to dump the report files
• componentTestUrlFactory: A function that returns the url for the component test to run
• componentTestVisualPathFactory: A function that returns the path for visual tests to run in
• visualThreshold (--v): A value between 0 and 1 to present the threshold at which a visual test may pass or fail
• onFinish: A function that can be used to do some extra work after Muppeteer is teared down
• useDocker (--d): The option for telling Muppeteer to run Chrome in Docker to better deal with environmental inconsistencies (default)
• headless (--h): Determines whether Chrome will be launched in a headless mode (without GUI) or with a head (not applicable with useDocker)
• disableSandbox (--s): Used to disable the sandbox checks if not using SUID sandbox (not applicable with useDocker)
• executablePath (--e): The option to set the version of Chrome to use duning the tests. By default, it uses the bundled version (not applicable with useDocker)

Puppeteer Extensions

You can access the Puppeteer Extensions API with page.extensions in your tests.

Example test case

const container = '.todoapp';
const input = 'header input';
const listItem = '.todo-list li';
const firstItem = listItem + ':nth-of-type(1)';
const firstItemToggle = firstItem + ' .toggle';
const firstItemRemoveButton = firstItem + ' button';
const secondItem = listItem + ':nth-of-type(2)';
const todoCount = '.todo-count';

describeComponent({name: 'todomvc', url: 'http://localhost:3000'}, () => {
    describe('Add a todo item', async () => {
        it('typing text and hitting enter key adds new item', async () => {
            await page.waitForSelector(input);
            await page.type(input, 'My first item');
            await page.keyboard.press('Enter');
            await page.waitForSelector(firstItem);
            assert.equal(await page.extensions.getText(firstItem), 'My first item');
            await assert.visual(container);
        });
        it('clicking checkbox marks item as complete', async () => {
            await page.waitForSelector(firstItemToggle);
            await page.click(firstItemToggle);

            // something to break the tests
            // await page.addStyleTag({ content: '.header { padding-top: 50px; }'});

            await page.extensions.waitForNthSelectorAttributeValue(listItem, 1, 'class', 'completed');
            await assert.visual(container);
        });
        it('typing more text and hitting enter adds a second item', async () => {
            await page.type(input, 'My second item');
            await page.keyboard.press('Enter');
            await page.waitForSelector(secondItem);
            assert.equal(await page.extensions.getText(secondItem), 'My second item');
            await assert.visual(container);
        });
        it('hovering over first item shows x button', async () => {
            await page.hover(firstItem);
            await assert.visual(container);
        });
        it('clicking on first item x button removes it from the list', async () => {
            await page.click(firstItemRemoveButton);
            await page.extensions.waitForElementCount(listItem, 1);
            assert.equal(await page.extensions.getText(todoCount), '1 item left');
            await assert.visual(container);
        });
    });
});

Docker and test fixtures

Muppeteer uses Docker by default to run tests. This helps to avoid environmental differences that could affect the rendering of content on the page. You can opt out by configuring the useDocker (--d) option accordingly.

If you are hosting your test fixtures on a local web server, you'd typically set the URL in the test to something like http://localhost:3000. When using Docker, the localhost will refer to the container, not the host machine. The simplest solution would be to reference the local IP in the test instead. For example, http://192.168.0.4:3000.

However, this breaks down when you are running on a device you don't know how to address, e.g. a cloud CI agent. To solve this problem, you can use the componentTestUrlFactory function in launch configuration to generate the URL. You can lookup the IP address of the current host and pass that through. This is used to run the example tests in this repo. See run-tests for an example.

...
componentTestUrlFactory: () => `http://${IP}:${PORT}`
...

Passing test output

Failing test output

Understanding visual failures

Baseline image

This is the visual that is versioned in your app repo. It is the source of truth.

Current image

This is the screenshot taken during the test. In this example, we can see that some padding has pushed the text input field down.

Difference image

This is an image showing where the differences are. Each difference is layered on top of one another. Here we can see that the "What needs to be done?" placeholder has moved down, and so it's sitting on top of where "My first item" previously was.