@empiricalrun/test-run

test-run

1. Introduction

@empiricalrun/test-run is a CLI tool (and library) designed to facilitate running Playwright (or Appwright) tests in a project. It allows you to:

• Run all tests or a specific test.
• Filter tests by name, file, or suites.
• Filter Playwright projects to run.
• Skip teardown test blocks.
• Integrate with an external dashboard for environment and build management.
• Forward additional arguments directly to the underlying Playwright/Appwright test runner.

2. Installation

Install from npm (usually as part of an Empirical-run monorepo, but can be done standalone):

npm install -D @empiricalrun/test-run

You can also use the CLI directly with npx.

3. Basic Usage

The CLI entry point is defined in package.json under "bin". The CLI provides three subcommands:

npx @empiricalrun/test-run run [flags] [Playwright/test arguments]  # Run tests (default)
npx @empiricalrun/test-run merge [flags]                            # Merge blob reports
npx @empiricalrun/test-run failed-list <run-id> [flags]             # Build test list from failed run

The run command is the default, so you can omit it:

npx @empiricalrun/test-run [flags] [Playwright/test arguments]

Example (running a test named "foo"):

npx @empiricalrun/test-run -n "foo" --retries 0

Here, --retries 0 (and any unrecognized options) get passed on to the underlying Playwright command.

4. CLI Flags

run command

Below are the main CLI options for the run command (implemented in src/bin/commands/run.ts):

Flag/Option	Description
-n, --name <test-name>	The scenario name (title) of the specific test to run. If provided, only that particular test/scenario is run instead of all tests.
-d, --dir <test-dir>	Directory containing tests. Defaults to tests.
-f, --file <test-file-path>	Specific test file path. If provided along with --name, it targets that file for the named test.
-p, --project <project-name...>	One or more (space-delimited) project names/patterns used to filter the test run. Examples: --project web or --project ios desktop.
-s, --suites <suites>	A comma-separated list of suite or describe-block names. Example: --suites "suiteA,suiteB".
--skip-teardown	Skips teardown tests by marking them as “skipped.” Reverts changes when the process exits.
--token <token>	A base64-encoded token that, when decoded, can define test configuration (such as a test list).
--forbid-only	Fails the run if there are any .only blocks in the code.

Other unrecognized flags on the command line will be appended as Playwright arguments.

merge command

Merges blob reports from sharded test runs:

Flag/Option	Description
-b, --blob-dir <path>	Path to the blob-report directory containing sharded reports.
-c, --cwd <path>	Working directory for the merge operation.

npx @empiricalrun/test-run merge --blob-dir ./blob-reports --cwd ./output

failed-list command

Builds a test list from a failed test run for retrying:

Flag/Option	Description
<run-id>	(Required) Test run ID to fetch failed tests from.
-o, --output <path>	Output file path for the test list.
-r, --repo-name <name>	Repository name for project lookup.
-p, --repo-path <path>	Path to local repo to expand serial blocks.
-v, --verbose	Enable verbose logging.

npx @empiricalrun/test-run failed-list abc123 --output ./failed-tests.json

5. Environment Variables

The following environment variables can further customize behavior:

• PROJECT_NAME
  Overrides project name detection from package.json.

• TEST_RUN_ENVIRONMENT
  This helps fetch playwright projects, environment details and potential build artifacts before tests start.

• BUILD_URL
  Overrides the build URL for downloading any build artifacts.

6. Running group of tests vs. All Tests

• Group Test:
  Use --name to identify one tests with the specfied name, optionally adding --file or --suites if needed.

  npx @empiricalrun/test-run --name "my test" --file tests/example.spec.ts
  

• All Tests:
  Omit --name. This runs all tests.

  npx @empiricalrun/test-run --reporter=list
  

7. Skipping Teardown Tests

If you pass --skip-teardown, the CLI will:

• Temporarily annotate *.teardown.ts files so that they are skipped (for example, converting teardown calls to teardown.skip).
• After the run, it automatically reverts the code changes, triggered by process exit signals.

This is useful for omitting teardown steps during certain runs or local development.

9. Environment & Build (Optional)

If TEST_RUN_ENVIRONMENT is defined:

• The CLI fetches environment details (and optionally a build artifact) via the remote “dashboard” (see dashboard.ts).
• If no custom BUILD_URL is set, it uses the environment’s latest build.
• It runs an npm script named download to grab the build.

This flow makes sure tests run against the relevant environment or a fresh build artifact.

10. Examples

• Run a single named test

  npx @empiricalrun/test-run -n "login test"
  

• Use specific test file and name

  npx @empiricalrun/test-run -n "logout test" -f tests/logout.spec.ts
  

• Run multiple projects and skip teardown

  npx @empiricalrun/test-run -n "mobile login" --project android --project ios --skip-teardown
  

• Run all tests with Playwright arguments

  npx @empiricalrun/test-run -- --worker=2 --reporter=line
  

11. Troubleshooting & Notes

• The CLI automatically appends unrecognized flags to the underlying playwright test (or appwright test) command.
• Errors like “No test block found” often mean the CLI did not locate your test name or file. Double-check --name, --file, etc.
• The teardown skip logic modifies files but reverts them on exit. If a process is forcibly killed, you may need to manually restore.
• If environment fetching or build downloading fails, confirm your environment slug and project name match those recognized by your dashboard.

14. Conclusion

@test-run simplifies orchestrating Playwright tests, especially for:

• Filtering or focusing on specific test scenarios.
• Managing environment-based test runs.
• Skipping teardown blocks.
• Working with multiple or specialized Playwright projects.

Explore the source (particularly src/bin/commands/ and src/utils) for deeper technical insights and advanced customization.