muppeteer

Muppeteer

Muppeteer is a visual regression testing framework for running UI tests in Chrome. It's composed of a number of existing Node modules:

• Mocha - a test runner framework
• Chai - an assertion library
• Puppeteer - a library for interacting with Chrome and web pages
• Pixelmatch - a pixel-level image comparison library

In addition, it provides the following core features:

• Visual Regression Testing - a screenshot-based image comparison module that hooks onto the assertion API. Read on for more discussion on this.
• Test Interface - a modification of Mocha's BDD interface with built-in browser setup steps and other user configurable hooks
• Test Launcher - a CLI and configuration function for launching test suites
• Page API - the API for Muppeteer's page interactions

Muppeteer's main goal is to abstract the, often, tedious boilerplate setup code needed to write tests with Puppeteer, and provide a convenient API for testing UI functionality. It was inspired by the PhantomCSS and CasperJS libraries.

This framework is in beta pre-release. While in beta, it is subject to breaking changes. There is also little test coverage as of yet. This is in progress. It is not recommended for use in production while in beta.

• Configuration

  • CLI

  • Configuration function

• API

• Example test case

  • Passing test output

  • Failing test output

  • Understanding visual failures

Configuration

You can configure Muppeteer via the CLI or a configuration function

CLI

The CLI script can be referenced at lib/test-launcher-cli.

It is run like node <<path-to-muppeteer>>/lib/test-launcher-cli <<args>>

Example

 "scripts": {
    "test": "node node_modules/muppeteer/lib/test-launcher-cli --t tests --f test.js --r tests/report"
  }

See Options

Configuration function

The configuration can be referenced at src/test-launcher.

Example

const ConfigureLauncher = require('../src/test-launcher');
const path = require('path');
const testsPath = path.join(__dirname, 'tests');

ConfigureLauncher({
        testDir: testsPath,
        testFilter: 'test.js',
        reportDir: `${testsPath}/report`,
        visualThreshold: 0.05,
        headless: true,
        disableSandbox: false,
        onFinish: () => {
            // do something after the tests have complete
        }
    }
).launch();

Options

Note: Only options with -- can be run with the proceeding flag in the CLI interface.

• testDir (--t): The directory for Mocha to look for the test files
• testFilter (--f): Allows you to pass in some text to filter the test file name, it's just a substring match, nothing fancy
• shouldRebaseVisuals (--b): A flag to tell the visual regression engine to replace the existing baseline visuals
• reportDir (--r): The directory for the Mocha reporter to dump the report files
• componentTestUrlFactory: A function that returns the url for the component test to run
• componentTestVisualPathFactory: A function that returns the path for visual tests to run in
• visualThreshold (--v): A value between 0 and 1 to present the threshold at which a visual test may pass or fail
• onFinish: A function that can be used to do some extra work after Muppeteer is teared down
• headless (--h): Determines whether Chrome will be launched in a headless mode (without GUI) or with a head
• disableSandbox (--s): Used to disable the sandbox checks if not using SUID sandbox
• executablePath (--e): The option to set the version of Chrome to use duning the tests. By default, it uses the bundled version

Example test case

const container = '.todoapp';
const input = 'header input';
const listItem = '.todo-list li';
const firstItem = listItem + ':nth-of-type(1)';
const firstItemToggle = firstItem + ' .toggle';
const firstItemRemoveButton = firstItem + ' button';
const secondItem = listItem + ':nth-of-type(2)';
const todoCount = '.todo-count';

describeComponent({name: 'todomvc', url: 'http://localhost:3000'}, () => {
    describe('Add a todo item', async () => {
        it('typing text and hitting enter key adds new item', async () => {
            await page.waitForSelector(input);
            await page.type(input, 'My first item');
            await page.keyboard.press('Enter');
            await page.waitForSelector(firstItem);
            assert.equal(await page.getText(firstItem), 'My first item');
            await assert.visual(container);
        });
        it('clicking checkbox marks item as complete', async () => {
            await page.waitForSelector(firstItemToggle);
            await page.click(firstItemToggle);

            // something to break the tests
            // await page.addStyleTag({ content: '.header { padding-top: 50px; }'});

            await page.waitForNthSelectorAttributeValue(listItem, 1, 'class', 'completed');
            await assert.visual(container);
        });
        it('typing more text and hitting enter adds a second item', async () => {
            await page.type(input, 'My second item');
            await page.keyboard.press('Enter');
            await page.waitForSelector(secondItem);
            assert.equal(await page.getText(secondItem), 'My second item');
            await assert.visual(container);
        });
        it('hovering over first item shows x button', async () => {
            await page.hover(firstItem);
            await assert.visual(container);
        });
        it('clicking on first item x button removes it from the list', async () => {
            await page.click(firstItemRemoveButton);
            await page.waitForElementCount(listItem, 1);
            assert.equal(await page.getText(todoCount), '1 item left');
            await assert.visual(container);
        });
    });
});

Passing test output

Failing test output

Understanding visual failures

Baseline image

This is the visual that is versioned in your app repo. It is the source of truth.

Current image

This is the screenshot taken during the test. In this example, we can see that some padding has pushed the text input field down.

Difference image

This is an image showing where the differences are. Each difference is layered on top of one another. Here we can see that the "What needs to be done?" placeholder has moved down, and so it's sitting on top of where "My first item" previously was.