@testorim/cli

Testorim CLI

Run AI browser QA tests from your terminal, your CI pipeline, your coding agent (MCP) or your own test code (a JavaScript and TypeScript library).

Monorepo: Repository README · Backend / API: packages/backend

npm install -g @testorim/cli
testorim login
testorim list
testorim run <procedure-id>

Quick start

• Generate an API key at https://app.testorim.com/settings/api-keys.

• Sign in:

  testorim login
  

  The key is saved to ~/.testorim/config.json with 0600 perms. You can also pass it via the TESTORIM_API_KEY env var (preferred for CI).

• Run a saved procedure:

  testorim run <procedure-id>
  

  Exits 0 on pass, 1 on fail, 2 on error, 3 when the run needs review, so CI can gate on it.

Ad-hoc tests

testorim run \
  --project <project-id> \
  --description "Sign in with [e-mail] and verify the dashboard loads"

Override the project's base URL for a single run (e.g. point at a PR preview):

testorim run <procedure-id> --base-url https://preview-42.staging.example.com

Negative tests

testorim run \
  --project <project-id> \
  --description "Try to log in with the wrong password and verify the error" \
  --expectation fail

The --expectation flag frames the report: fail means the run "passes" when the app correctly rejects.

Use it from code

The package is also a JavaScript and TypeScript library, so a test suite can run Testorim tests and gate on the verdict.

npm install --save-dev @testorim/cli

Node 20 or newer. The package is ESM, ships its own types and has no dependencies. Create an API key in Testorim under Settings, API keys and set TESTORIM_API_KEY=tst_live_....

import { Testorim } from "@testorim/cli";

const testorim = new Testorim(); // reads TESTORIM_API_KEY

const run = await testorim.runTest(
  "shop.example.com", // a project's name, address or id
  "Add the Blue Top to the cart, open the cart and check that it shows 1 item.",
);
console.log(run.summary());
run.assertPassed(); // throws TestorimRunFailed unless the verdict is passed

runTest starts the run and waits for the verdict, polling every 3 seconds for up to 10 minutes. Name the buttons, fields and text as they appear on the page.

More of the client:

await testorim.projects();                                  // [{ id, name, baseUrl }, ...]
await testorim.createProject("https://shop.example.com");   // the existing project if the address is already tested
await testorim.tests("Shop");                               // the project's saved tests

// Replay a saved test: by name within a project, or by id
await testorim.runSaved("Checkout", { project: "Shop" });
await testorim.runSaved("5f0c...");

// A negative test passes when the app refuses what the description tries
await testorim.runTest("Shop", "Sign in with a wrong password and check that an error is shown.", { expectFailure: true });

// Run against another address, for example a pull request's preview
await testorim.runTest("Shop", "...", { baseUrl: "https://pr-42.preview.shop.example.com" });

// Start without waiting, then wait, read or stop it
let run = await testorim.runTest("Shop", "...", { wait: false });
run = await testorim.waitFor(run.id, { timeoutMs: 900_000, onStatus: (r) => console.log(r.status) });
run = await testorim.getRun(run.id);
await testorim.cancel(run.id);

When the wait runs out, runTest, runSaved and waitFor return the run as it is (run.done is false) instead of throwing. Pass raiseOnTimeout: true to get TestorimTimeout instead. new Testorim({ apiKey, apiUrl, baseUrl, requestTimeoutMs }) overrides the key, the API host (default TESTORIM_API_URL, then https://app.testorim.com), the address every run opens, and the per-request timeout (30 seconds).

The site has to be reachable from the internet: Testorim's browsers refuse localhost and private addresses. To test work in progress, run against a preview deployment or expose your dev server through a tunnel. Every run counts against your workspace's plan.

Playwright Test

import { test } from "@playwright/test";
import { Testorim } from "@testorim/cli";

const testorim = new Testorim();

test("checkout works", async () => {
  test.setTimeout(11 * 60_000); // a run takes minutes; Playwright's default is 30 seconds
  const run = await testorim.runSaved("Checkout", { project: "Shop" });
  run.assertPassed();
});

A failed run fails the test with its summary:

TestorimRunFailed: Verdict: FAILED (1 passed, 1 failed, 1 skipped, 41s)
Run: https://app.testorim.com/runs/...
Failed step 2: click "Create account": The page showed Something went wrong [the app did not behave as described]

Vitest works the same way; give the test a timeout: test("checkout works", async () => { ... }, 11 * 60_000). Jest takes the same third argument; since the package is ESM, run Jest with its ESM support (NODE_OPTIONS=--experimental-vm-modules npx jest). In a CommonJS project, require("@testorim/cli") works on Node 20.19 or newer and 22.12 or newer.

The run

Property	What
id, url	The run's id and its page in Testorim
status	pending, running, completed, failed or cancelled
verdict	passed, failed or needs_review once finished; null while it runs and for a cancelled run
passed, failed, needsReview, cancelled, done	true or false
passedCount, failedCount, skippedCount, durationSeconds	Step counts and how long the steps took
report	The written report, in Markdown
steps, failedSteps	The steps, and the failed ones: number (from 1), action, target, error, blame, blameText, unconfirmed
startUrl, finalUrl	Where the run started and ended
videoUrl, traceUrl, screenshotUrl	The recording, the Playwright trace and the final screenshot. These links expire an hour after they were read; getRun gives fresh ones
summary()	Readable lines: verdict, counts, why, each failed step and the run's page
assertPassed()	Returns the run if it passed, else throws TestorimRunFailed with the summary

summary() leaves the evidence links out, because they open without signing in and the summary often lands in CI logs.

Verdicts and blame

Verdict	Meaning
passed	Every step passed and the report confirms the request was met
failed	A step failed. Each failed step says who was at fault (below)
needs_review	Every step passed, but the report could not confirm the request was met. summary() gives the reason
null	The run was cancelled, or has not finished

blame	Meaning
app	The app did not behave as described: a real finding
test	The test could not do what it described; check the wording against the page
unsupported	The check asked for is not supported
internal	A Testorim internal error, not your app

unconfirmed is true when a step found text that differs from what was expected but cannot tell whether the page or the expected text is wrong. assertPassed() throws for every verdict but passed, so a run that needs review stops CI for a person. To let it through, check run.failed yourself.

Errors

Every error derives from TestorimError, and each is exported.

Error	When	Properties
AuthenticationError	401: the API key is missing, wrong, revoked or expired	status, body
RefusedError	402, 429 or 503: Testorim will not start the run now	code, upgradeUrl, retryable, retryAfter
NotFoundError	404, or no project or saved test matches the name you gave	code (other_workspace when the key's owner has it in another workspace)
ApiError	Any other error status, such as 400, 403 (read_only_role, api_key_scope) or 500. The three above derive from it	status, body, code
TestorimUnreachable	The API could not be reached; the message names the host
TestorimRunFailed	assertPassed() on a run that did not pass	run
TestorimTimeout	A wait with raiseOnTimeout: true ran out	run
TestorimError	No API key, or a name that matches more than one project or saved test

RefusedError.code is onboarding_exhausted or locked (402: pay to go on, upgradeUrl is the pricing page), plan_quota_exhausted or concurrency_limit (429: retry once the period renews or a run finishes) or service_busy (503: Testorim's own capacity, retry). A 429 with no code is a rate limit; retryAfter says how many seconds to wait. The client never retries a trigger by itself: a retried trigger is a second run.

Use it from a coding agent (MCP)

testorim mcp is a Model Context Protocol server on stdio. Any agent that speaks MCP can then run a Testorim test and read the verdict: Claude Code, Codex, Cursor, VS Code (Copilot), Windsurf, Antigravity, Gemini CLI, Claude Desktop and others.

It gives the agent six tools:

Tool	What it does
list_projects	The projects (sites under test) in your workspace
list_tests	A project's saved tests
run_test	Runs a saved test, or any test described in plain English, and waits for the verdict: which steps failed, whether the app or the test was at fault, the report, and links to the video and trace
get_run	Reads a run's result
cancel_run	Stops a run
create_project	Adds a site to test

Agents start MCP servers without your shell's environment, so put the key in the agent's config. Create a key in Testorim under Settings, API keys.

Claude Code

claude mcp add testorim -e TESTORIM_API_KEY=tst_live_... -- npx -y @testorim/cli mcp

Codex (~/.codex/config.toml)

[mcp_servers.testorim]
command = "npx"
args = ["-y", "@testorim/cli", "mcp"]
env = { TESTORIM_API_KEY = "tst_live_..." }

Cursor (.cursor/mcp.json), Windsurf (~/.codeium/windsurf/mcp_config.json), Antigravity (Manage MCP servers, View raw config), Gemini CLI (~/.gemini/settings.json) and Claude Desktop (claude_desktop_config.json) all take the same block:

{
  "mcpServers": {
    "testorim": {
      "command": "npx",
      "args": ["-y", "@testorim/cli", "mcp"],
      "env": { "TESTORIM_API_KEY": "tst_live_..." }
    }
  }
}

VS Code (.vscode/mcp.json)

{
  "servers": {
    "testorim": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@testorim/cli", "mcp"],
      "env": { "TESTORIM_API_KEY": "tst_live_..." }
    }
  }
}

Then ask the agent something like: "Deploy a preview, then use Testorim to check that a new user can sign up and reach the dashboard."

The site has to be reachable from the internet: Testorim's browsers refuse localhost and private addresses. To test work in progress, run it against a preview deployment (run_test takes a baseUrl) or expose your dev server through a tunnel. Add "TESTORIM_API_URL" to env if your workspace lives on another host.

CI usage (GitHub Actions example)

- name: Run smoke test
  env:
    TESTORIM_API_KEY: ${{ secrets.TESTORIM_API_KEY }}
  run: npx @testorim/cli run <procedure-id>

Flags

Flag	What
--api-key <key>	Override stored key
--api-url <url>	Override API base URL (default https://app.testorim.com)
--project <id>	Project id (required for ad-hoc tests, optional otherwise)
--description "..."	Plain-English test prompt for ad-hoc runs
--base-url <url>	Override the project's base URL for this run only
--expectation <pass|fail|unknown>	Test intent (default pass)
--json	Machine-readable output
--quiet	Suppress progress lines

Exit codes

Code	Meaning
0	Run passed (overallStatus = "passed")
1	Run failed (overallStatus = "failed")
2	Couldn't run or no verdict (auth error, bad input, a refusal, a cancelled run, server error, timeout)
3	Needs review (overallStatus = "needs_review"): every step passed, but the report could not confirm the request was met

Configuration

Precedence (highest first):

• CLI flags
• Environment variables: TESTORIM_API_KEY, TESTORIM_API_URL
• ~/.testorim/config.json