@open-xchange/codecept-runner

@open-xchange/codecept-runner

A runner script for E2E tests. Supports parallel execution, rerunning failed tests, and continuation.

Help

The switch --help (or -h) prints an overview with all supported switches. Example:

yarn codecept-runner --help

CodeceptJS Configuration File

The runner needs to know the location of the configuration file codecept.conf.js. If it is not located in the project root directory, the switch --config (or -c) allows to specify an alternative path.

Example:

yarn codecept-runner --config=./e2e/codecept.conf.js

Filter by Test Titles

The tests to be executed can be selected by full test titles (including scenario names and tags) using the switches --grep (or -g) and --ignore (or -i). Both switches expect a regular expression.

• If --grep is omitted, all existing tests will be selected.
• If --ignore is omitted, no selected test will be skipped.
• If --grep and --ignore are specified, the tests selected by both will be skipped.

Example:

yarn codecept-runner --grep=@stable --ignore=@experimental

will execute all tests with the tag "stable" that do not contain the tag "experimental".

Retry Failed Tests

Failed tests can be rerun automatically using the switch --retries (or -r). The number after the switch specifies the number of additional test runs after the initial test run.

Example:

$ yarn codecept-runner --retries=2
running 1000 tests ...
20 tests failed

retrying 20 tests ...
2 tests failed

retrying 2 tests ...
all tests passed

Parallel Execution

Tests will be executed in parallel with multiple workers.

By default, the initial number of workers is half of the number of processors available on the system. For every retry run, the number of workers will be halved to compensate for flaky tests caused by system load.

Example:

$ yarn codecept-runner --retries=2
running 1000 tests with 8 workers ...
20 tests failed

retrying 20 tests with 4 workers ...
3 tests failed

retrying 3 tests with 2 workers ...
all tests passed

The number of workers can be changed using the switch --workers (or -w). The additional switch --retry-workers (or uppercase -W) allows to set a different number of workers for retrying failed tests.

Example:

$ yarn codecept-runner --workers=4 --retries=1 --retry-workers=3
running 1000 tests with 4 workers ...
20 tests failed

retrying 20 tests with 3 workers ...
all tests passed

Continuation

The results of a test run will be saved in a JSON file in the configured output directory. After a test run finishes with failed tests, or has been interrupted, the remaining tests (failed and pending) can be restarted using the switch --continue (or uppercase -C).

Example:

$ yarn codecept-runner --grep@stable
running 1000 tests ...
20 tests failed

$ yarn codecept-runner --continue
running 20 tests ...
all tests passed

Dry Run

Using the switch --dry (or -d) the runner will print log messages but will not actually run any test.

Example:

$ yarn codecept-runner --workers=4 --retries=1 --dry
running 1000 tests with 4 workers ...
1000 tests skipped