@shiplightai/cli

@shiplightai/cli

Run .test.yaml files as first-class Playwright tests. YAML tests run alongside your existing .test.ts files with npx playwright test — no separate CLI or config needed.

Quick Start

1. Install

npm install -D @shiplightai/cli @shiplightai/sdk-pro @playwright/test

2. Configure

In your playwright.config.ts:

import { defineConfig } from '@playwright/test';
import { shiplightConfig } from '@shiplightai/cli';

export default defineConfig({
  ...shiplightConfig(),

  testDir: './tests',
  use: {
    headless: true,
    viewport: { width: 1280, height: 720 },
  },
});

3. Write a YAML test

Create tests/login.test.yaml:

goal: Verify user can log in
url: https://example.com/login

statements:
  - Click on the username field and type "testuser"
  - Click on the password field and type "secret123"
  - Click the Login button
  - "VERIFY: Dashboard page is visible"

4. Run

npx playwright test

Playwright discovers both *.test.ts and *.test.yaml files. YAML files are transparently transpiled to .yaml.spec.ts files next to the source (cached by mtime).

5. Gitignore generated files

Add to your .gitignore:

*.yaml.spec.ts

How It Works

shiplightConfig() runs during Playwright config loading and:

• Scans for **/*.test.yaml files
• Transpiles each to a *.yaml.spec.ts file next to the source
• Skips files that haven't changed (mtime-based cache, includes template dependencies and transpiler version)

Playwright then discovers the generated .yaml.spec.ts files through its default testMatch pattern. If you override testMatch, make sure it includes *.spec.ts.

YAML Test Format

Basic Structure

goal: Description of what this test verifies
url: https://your-app.com/starting-page

statements:
  - Step described in natural language
  - Another step
  - "VERIFY: Expected outcome"

teardown:
  - Clean up step

Field	Required	Description
goal	Yes	Test description (used as the Playwright test name)
url	Yes	Starting URL to navigate to
statements	Yes	List of test steps
teardown	No	Steps that always run after the test (like finally)

Statement Types

Draft (natural language)

Plain strings are AI-resolved steps. The agent figures out what to click, type, etc.

statements:
  - Navigate to the settings page
  - Click the "Delete Account" button
  - Type "confirm" in the confirmation dialog

VERIFY

Asserts a condition using AI. Must be a quoted string prefixed with VERIFY:.

statements:
  - "VERIFY: The success message is displayed"
  - "VERIFY: User is redirected to the dashboard"

ACTION (with action entity)

Deterministic actions with explicit locators. These replay fast (~1s) without AI.

statements:
  - description: Click the Submit button
    action_entity:
      action_description: Click the Submit button
      locator: "getByRole('button', { name: 'Submit' })"
      action_data:
        action_name: click
        kwargs: {}

  - description: Type email address
    action_entity:
      action_description: Type email address
      locator: "getByLabel('Email')"
      action_data:
        action_name: input_text
        kwargs:
          text: "[e-mail]"

  - description: Press Enter
    action_entity:
      action_data:
        action_name: press
        kwargs:
          keys: Enter

STEP (grouping)

Groups related statements under a label.

statements:
  - STEP: Fill in the registration form
    statements:
      - Type "John" in the first name field
      - Type "Doe" in the last name field
      - Type "[e-mail]" in the email field

IF / ELSE (conditional)

Conditional branching with AI or JavaScript conditions.

statements:
  # AI condition
  - IF: A cookie consent banner is visible
    THEN:
      - Click the Accept button
    ELSE:
      - "VERIFY: No banner is blocking the page"

  # JavaScript condition
  - IF: "JS: page.url().includes('/login')"
    THEN:
      - Enter credentials and log in

WHILE loop

Repeating steps with a timeout.

statements:
  - WHILE: There are more items in the list
    DO:
      - Click the next item
      - "VERIFY: Item details are shown"
    timeout_ms: 30000

Supported Actions

Actions used in action_entity.action_data.action_name:

Action	kwargs	Description
click	—	Click an element
double_click	—	Double-click an element
right_click	—	Right-click an element
hover	—	Hover over an element
input_text	text	Type text into an input
clear_input	—	Clear an input field
press	keys	Press a keyboard key (e.g., Enter, Tab)
send_keys_on_element	keys	Press a key on a specific element
select_dropdown_option	text	Select a dropdown option by text
scroll	down, num_pages	Scroll the page
scroll_to_text	text	Scroll to text on the page
go_to_url	url, new_tab	Navigate to a URL
go_back	—	Browser back
reload_page	—	Reload the page
wait	seconds	Wait for a duration
wait_for_page_ready	—	Wait for page load
verify	statement or code	Assert a condition (AI or JS)
js_code	code	Run inline JavaScript
function	functionName, parameterNames, parameterValues	Call a function
switch_tab	tab_index	Switch browser tab
close_tab	—	Close current tab
upload_file	file_path	Upload a file
save_variable	name, value	Save a variable for later use

Locators

Action entities can specify element locators in two ways:

# Playwright locator (preferred)
locator: "getByRole('button', { name: 'Submit' })"

# XPath
xpath: "//button[@id='submit']"

If both are present, locator takes priority. If neither is present, the AI agent resolves the element from the action description.

Frames

For elements inside iframes:

action_entity:
  frame_path:
    - "iframe#main"
  locator: "getByText('Hello')"
  action_data:
    action_name: click
    kwargs: {}

Extensions

Custom Test Name

Override the Playwright test name (defaults to goal):

name: Login with valid credentials
goal: Verify login flow works
url: https://example.com
statements:
  - ...

Tags

Add Playwright tags for filtering with --grep:

tags:
  - smoke
  - auth
goal: Login test
url: https://example.com
statements:
  - ...

Run: npx playwright test --grep @smoke

Playwright Fixtures

Pass options to test.use():

use:
  viewport:
    width: 375
    height: 812
  locale: fr-FR
goal: Mobile French layout
url: https://example.com
statements:
  - ...

Variables

Use {{VAR_NAME}} to reference environment variables. At transpile time, these become process.env.VAR_NAME references in the generated code.

statements:
  - description: Type username
    action_entity:
      locator: "getByLabel('Username')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{TEST_USER}}"

Set variables when running:

TEST_USER=admin TEST_PASS=secret npx playwright test

Templates

Extract reusable flows into template files and include them with template:.

Template file (templates/login.yaml):

params:
  - username
  - password

statements:
  - description: Enter username
    action_entity:
      locator: "getByLabel('Username')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{username}}"
  - description: Enter password
    action_entity:
      locator: "getByLabel('Password')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{password}}"
  - description: Click login
    action_entity:
      locator: "getByRole('button', { name: 'Log in' })"
      action_data:
        action_name: click
        kwargs: {}

Using the template:

goal: Purchase flow
url: https://example.com

statements:
  - template: ../templates/login.yaml
    params:
      username: "{{TEST_USER}}"
      password: "{{TEST_PASS}}"
  - Navigate to the checkout page
  - "VERIFY: Order summary is displayed"

Template params ({{username}}) are substituted at transpile time. Environment variables ({{TEST_USER}}) pass through to the generated code for runtime resolution.

Templates can be nested (max depth: 5) and circular references are detected.

Custom Functions

Call TypeScript functions from YAML using the function action with file#export syntax:

statements:
  - description: Seed test data
    action_entity:
      action_data:
        action_name: function
        kwargs:
          functionName: "../helpers/seed.ts#createTestUser"
          parameterNames:
            - page
            - email
          parameterValues:
            - page
            - "[e-mail]"

This generates:

import { createTestUser } from '../helpers/seed';
// ...
await createTestUser(page, "[e-mail]");

Configuration Options

shiplightConfig({
  // Directory to scan for .test.yaml files (default: process.cwd())
  scanDir: './tests',

  // API key for cloud features (optional)
  apiKey: process.env.SHIPLIGHT_API_KEY,
})

Agent Fixture

The package exports a custom test object with an agent fixture. Generated YAML tests use this automatically:

import { test, expect } from '@shiplightai/cli';

test('my test', async ({ page, agent }) => {
  // `agent` is a pre-configured WebAgent instance
  // `page` is the standard Playwright page
});

You can also use the fixture in your hand-written .test.ts files to get the same agent instance:

// tests/custom.test.ts
import { test, expect } from '@shiplightai/cli';

test('custom test with agent', async ({ page, agent }) => {
  await page.goto('https://example.com');
  await agent.run(page, 'Click the login button', 'step-1');
  await agent.assert(page, 'User is on the dashboard', 'step-2');
});

Authentication

Most apps require login before tests can interact with them. The recommended approach uses Playwright's built-in globalSetup to authenticate once, then share the session across all test workers via storageState.

1. Create e2e/global-setup.ts

This runs once before all tests. It uses @shiplightai/sdk-pro's WebAgent.loginPage() for AI-driven login — no fragile selectors that break when your login UI changes.

import { chromium, type FullConfig } from '@playwright/test';
import { readFile, mkdir } from 'fs/promises';

const AUTH_DIR = '.auth';
const STORAGE_STATE_PATH = `${AUTH_DIR}/storage-state.json`;

async function loadStorageState(): Promise<any | undefined> {
  try {
    const data = await readFile(STORAGE_STATE_PATH, 'utf-8');
    return JSON.parse(data);
  } catch {
    return undefined;
  }
}

async function globalSetup(config: FullConfig) {
  const baseURL = config.projects[0]?.use?.baseURL
    || process.env.PLAYWRIGHT_BASE_URL
    || 'https://your-app.com';

  const username = process.env.SHIPLIGHT_LOGIN_EMAIL;
  const password = process.env.SHIPLIGHT_LOGIN_PASSWORD;
  const loginUrl = process.env.SHIPLIGHT_LOGIN_URL;
  const totpSecret = process.env.SHIPLIGHT_LOGIN_TOTP_SECRET;

  if (!username || !password || !loginUrl) {
    console.log('[global-setup] Skipping login. Set SHIPLIGHT_LOGIN_EMAIL, SHIPLIGHT_LOGIN_PASSWORD, and SHIPLIGHT_LOGIN_URL to enable.');
    return;
  }

  const { WebAgent, createAgentContext, configureSdk, VariableStore } = await import('@shiplightai/sdk-pro');

  configureSdk({
    env: {
      GOOGLE_API_KEY: process.env.GOOGLE_API_KEY ?? '',
      ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY ?? '',
    },
  });

  const agent = new WebAgent(createAgentContext({
    model: process.env.SHIPLIGHT_MODEL || 'gemini-2.5-pro',
    variableStore: new VariableStore(),
  }));

  const browser = await chromium.launch();

  // Reuse cached storage state if available — the agent detects
  // "already signed in" and skips login automatically.
  const storageState = await loadStorageState();
  const context = await browser.newContext({
    baseURL,
    ...(storageState && { storageState }),
  });
  const page = await context.newPage();

  const absoluteLoginUrl = loginUrl.startsWith('http')
    ? loginUrl
    : `${baseURL}${loginUrl}`;

  const result = await agent.loginPage(page, {
    site_url: absoluteLoginUrl,
    num_verification_exprs: 0,
    account: {
      type: 'password',
      username,
      password,
      ...(totpSecret && {
        two_factor_auth_config: { type: 'totp', data: totpSecret },
      }),
    },
  });

  if (!result.success) {
    await browser.close();
    throw new Error('[global-setup] Login failed.');
  }

  await mkdir(AUTH_DIR, { recursive: true });
  await context.storageState({ path: STORAGE_STATE_PATH });
  await browser.close();
  console.log('[global-setup] Auth state saved.');
}

export default globalSetup;

2. Update playwright.config.ts

import { defineConfig } from '@playwright/test';
import { shiplightConfig } from '@shiplightai/cli';

export default defineConfig({
  ...shiplightConfig({ scanDir: './e2e' }),

  globalSetup: './e2e/global-setup.ts',
  testDir: './e2e',
  timeout: 120_000,
  use: {
    headless: true,
    viewport: { width: 1280, height: 720 },
    baseURL: process.env.PLAYWRIGHT_BASE_URL || 'https://your-app.com',
    storageState: '.auth/storage-state.json',
  },
});

3. Add to .gitignore

.auth/

4. Run with credentials

ANTHROPIC_API_KEY=sk-ant-... \
SHIPLIGHT_MODEL=claude-haiku-4-5 \
SHIPLIGHT_LOGIN_EMAIL=[e-mail] \
SHIPLIGHT_LOGIN_PASSWORD=yourpassword \
SHIPLIGHT_LOGIN_URL=/login \
npx playwright test

How it works

• globalSetup runs once before any test worker starts
• If .auth/storage-state.json exists and cookies are still valid, the agent detects "already signed in" and skips login
• If not signed in, the AI agent performs login (handles UI changes, 2FA, etc.)
• The resulting cookies/localStorage are saved to .auth/storage-state.json
• All test workers load this storage state — every test starts already authenticated

2FA / TOTP support

Set SHIPLIGHT_LOGIN_TOTP_SECRET to your TOTP secret and the agent will generate and enter the 2FA code automatically.

Environment Variables

Variable	Description
GOOGLE_API_KEY	Google AI API key (for Gemini models)
ANTHROPIC_API_KEY	Anthropic API key (for Claude models)
SHIPLIGHT_MODEL	AI model to use (default: gemini-2.5-pro)
SHIPLIGHT_LOGIN_EMAIL	Login email for globalSetup authentication
SHIPLIGHT_LOGIN_PASSWORD	Login password for globalSetup authentication
SHIPLIGHT_LOGIN_URL	Login page URL (absolute or relative to baseURL)
SHIPLIGHT_LOGIN_TOTP_SECRET	TOTP secret for 2FA (optional)
PLAYWRIGHT_STARTING_URL	Override the starting URL for all tests

Project Structure

project/
├── playwright.config.ts
├── e2e/
│   ├── global-setup.ts             # AI-driven login (optional)
│   ├── auth/
│   │   ├── login.test.ts           # Regular Playwright test
│   │   ├── signup.test.yaml        # YAML test (checked in)
│   │   └── signup.yaml.spec.ts     # Generated (gitignored)
│   └── checkout/
│       ├── purchase.test.yaml
│       └── purchase.yaml.spec.ts
├── templates/
│   └── login.yaml                  # Reusable template
├── .auth/                          # Cached auth state (gitignored)
│   └── storage-state.json
└── .gitignore                      # includes *.yaml.spec.ts, .auth/