shiplightai

shiplightai

AI-powered end-to-end testing for Playwright. Write tests as YAML with natural-language steps, run them alongside your existing .test.ts files, get self-healing locators and a visual debugger. One package ships both the shiplight CLI and the library you wire into playwright.config.ts.

Full documentation: docs.shiplight.ai

Quick start

Scaffold a new test project in under a minute:

npx shiplightai@latest create ./my-tests
cd my-tests
cp .env.example .env            # then set one of GOOGLE_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY
npm install
npx playwright install chromium
npx shiplight test              # runs the scaffolded starter test

The scaffolder writes package.json, playwright.config.ts, .env.example, .gitignore, and a runnable tests/example.test.yaml that exercises a live site. Open shiplight-report/index.html after the run to see per-step screenshots, videos, and traces.

Or add Shiplight to an existing Playwright project:

npm install shiplightai

CLI

Every command is invoked via npx shiplight <command> from inside a project that has shiplightai installed.

Command	Purpose
shiplight create <path>	Scaffold a new test project
shiplight test [args]	Run your YAML and Playwright test suite (forwards all Playwright flags)
shiplight debug <file>	Launch the interactive visual debugger for a YAML test. The server picks a free port automatically; the chosen URL is printed on startup. Pass --port N for a stable URL (CI, bookmarks). Multiple concurrent shiplight debug invocations work without any port coordination.
shiplight report [folder]	Regenerate or merge HTML reports
shiplight transpile [glob]	Transpile YAML tests to .yaml.spec.ts (usually automatic). --strict fails while statements are still unenriched drafts (no captured action/js)
shiplight spec <topic>	Print an authoring reference: yaml (the test language spec) or actions (every action and its parameters)

Full reference with flags, options, and troubleshooting: docs.shiplight.ai/local/cli-reference.

Library

Wire up YAML support and the Shiplight reporter in playwright.config.ts:

import { defineConfig, shiplightConfig } from 'shiplightai';

export default defineConfig({
  ...shiplightConfig(),

  testDir: './tests',
  use: {
    headless: true,
    viewport: { width: 1280, height: 720 },
  },
});

shiplightConfig() returns a partial Playwright config that wires in YAML transpilation, auto-discovered .env files, and the HTML reporter. Spread it into your own defineConfig and override whatever you need.

Chrome extensions

Point extensionDir at an unpacked extension. Use Playwright's standard launchOptions.args for test-only Chromium switches, including capture-source selection flags:

export default defineConfig({
  ...shiplightConfig(),
  use: {
    extensionDir: './dist',
    headless: false,
    launchOptions: {
      args: ['--auto-select-screen-capture-source'],
    },
  },
});

Playwright does not include a Chrome action popup in context.pages(). Import Shiplight's test fixture and request extensionActionPopup to open the real toolbar popup and receive it as a standard Playwright Page:

import { test, expect } from 'shiplightai/fixture';

test('starts from the real extension popup', async ({ extensionActionPopup }) => {
  await extensionActionPopup.getByRole('button', { name: 'Start recording' }).click();
});

This requires Chrome 127 or newer because it uses chrome.action.openPopup(). The extension/profile/debugging transport switches remain fixture-owned and cannot be overridden through launchOptions.args.

For the YAML statement language, authentication, variables, templates, and custom functions: docs.shiplight.ai/local.

Environment variables

Set at least one LLM provider API key in .env before running tests:

GOOGLE_API_KEY=...
# or ANTHROPIC_API_KEY=sk-ant-...
# or OPENAI_API_KEY=sk-...

The AI model is auto-selected from the first key set. Override with WEB_AGENT_MODEL=<model>.

Shiplight uses an explicit env var allowlist: only the vars it forwards into its internal SDK config are accessible to the agent. Any env var not on the list is invisible to the SDK by design. Full list (OPENAI_BASE_URL, Vertex AI routing, Mailgun keys, etc.): docs.shiplight.ai/local/cli-reference#environment-variables.

Requirements

• Node.js >= 22
• @playwright/test 1.60.0 — declared as a peer dependency and installed automatically by npm install
• Chromium — install on demand with npx playwright install chromium

Links

• Documentation: docs.shiplight.ai
• VS Code extension: github.com/ShiplightAI/vscode-extension
• Cloud platform: shiplight.ai