@open-xchange/codecept-runner

@open-xchange/codecept-runner

A runner script for E2E tests. Supports parallel execution, rerunning failed tests, and continuation.

Help

The switch --help (or -h) prints an overview with all supported switches. Example:

pnpm codecept-runner --help

CodeceptJS Configuration File

The runner needs to know the location of the CodeceptJS configuration file. If it is not located in the project root directory, the switch --config (or -c) allows to specify an alternative path.

Like CodeceptJS itself, the runner will try all well-known names for the configuration file:

• codecept.config.js
• codecept.conf.js
• codecept.json
• codecept.config.ts
• codecept.conf.ts

Example:

# by exact file name
pnpm codecept-runner --config=./e2e/special.config.ts

# by path with auto-detection
pnpm codecept-runner --config=./e2e

Filter by Test Titles

The tests to be executed can be selected by full test titles (including scenario names and tags) using the switches --grep (or -g) and --ignore (or -i). Both switches expect a regular expression.

• If --grep is omitted, all existing tests will be selected.
• If --ignore is omitted, no selected test will be skipped.
• If --grep and --ignore are specified, the tests selected by both will be skipped.

Example:

pnpm codecept-runner --grep=@stable --ignore=@experimental

will execute all tests with the tag 'stable' that do not contain the tag 'experimental'.

Retry Failed Tests

Failed tests can be rerun automatically using the switch --retries (or -r). The number after the switch specifies the number of additional test runs after the initial test run.

Example:

$ pnpm codecept-runner --retries=2
running 1000 tests ...
20 tests failed

retrying 20 tests ...
2 tests failed

retrying 2 tests ...
all tests passed

Parallel Execution

Tests will be executed in parallel with multiple workers.

By default, the initial number of workers is half of the number of processors available on the system. For every retry run, the number of workers will be halved to compensate for shaky tests caused by system load.

Example (on a system with 16 reported processors):

$ pnpm codecept-runner --retries=2
running 1000 tests with 8 workers ...
20 tests failed

retrying 20 tests with 4 workers ...
3 tests failed

retrying 3 tests with 2 workers ...
all tests passed

The number of workers can be changed using the switch --workers (or -w). The additional switch --retry-workers (or uppercase -W) allows to set a different number of workers for retrying failed tests.

Example:

$ pnpm codecept-runner --workers=4 --retries=1 --retry-workers=3
running 1000 tests with 4 workers ...
20 tests failed

retrying 20 tests with 3 workers ...
all tests passed

The option --retry-workers supports the keywords same and halve instead of a number. The keyword same causes to use the initial number of workers for all retry runs. The keyword halve (which is the default) causes to halve the number of workers for every retry run.

Example (on a system with 16 reported processors):

$ pnpm codecept-runner --retries=1 --retry-workers=same
running 1000 tests with 8 workers ...
20 tests failed

retrying 20 tests with 8 workers ...
all tests passed

Partitioning

A chunk of all matching tests can be executed with the option --partition (or -P). Its value consists of the one-based chunk index and of the total number of chunks, separated with a forward slash.

$ pnpm codecept-runner --partition=2/10
found 1000 matching tests
running chunk 2 of 10 with 100 tests
all tests passed

Continuation

The results of a test run will be saved in a JSON file in the configured output directory. After a test run finishes with failed tests, or has been interrupted, the remaining tests (failed and pending) can be restarted using the switch --continue (or uppercase -C).

Example:

$ pnpm codecept-runner --grep=@stable
running 1000 tests ...
20 tests failed

$ pnpm codecept-runner --continue
running 20 tests ...
all tests passed

The path to the JSON file can be customized with the option --state-file (or uppercase -S).

Example:

$ pnpm codecept-runner --grep=@stable --state-file=./.temp/test-state.json
running 1000 tests ...
20 tests failed

$ pnpm codecept-runner --continue --state-file=./.temp/test-state.json
running 20 tests ...
all tests passed

Dry Run

Using the switch --dry-run (or -d) the runner will print log messages but will not actually run any test.

Example:

$ pnpm codecept-runner --workers=4 --retries=1 --dry-run
running 1000 tests with 4 workers ...
1000 tests skipped