@debugg-ai/cli
Advanced tools
The TypeScript/Node.js implementation of SystemEval - a unified evaluation framework providing objective, deterministic, and traceable test execution.
Homepage: debugg.ai | Docs: debugg.ai/docs/systemeval
Test results should be facts, not opinions.
Traditional test runners produce ambiguous output that requires human interpretation. Did the build pass? Sort of. Are we ready to deploy? Probably. SystemEval eliminates this ambiguity.
PASS, FAIL, or ERROR - nothing elseAgent writes code
↓
systemeval test --json (or: npx @debugg-ai/cli test)
↓
Structured result (pass/fail/error + metrics)
↓
Agent reads result, fixes failures
↓
Repeat until PASS
# Global CLI installation
npm install -g @debugg-ai/cli
# Project dependency (for programmatic usage)
npm install @debugg-ai/cli
# Set your API key
export DEBUGGAI_API_KEY=your_api_key_here
# Run tests on current git changes
debugg-ai test
# Test last 3 commits
debugg-ai test --last 3
# Test all commits in a PR individually
debugg-ai test --pr-sequence
# Wait for local development server
debugg-ai test --wait-for-server
import { runDebuggAITests } from '@debugg-ai/cli';
const result = await runDebuggAITests({
apiKey: process.env.DEBUGGAI_API_KEY!,
repoPath: process.cwd(),
waitForServer: true,
serverPort: 3000,
});
// Deterministic verdict
if (result.success) {
process.exit(0); // PASS
} else {
process.exit(1); // FAIL
}
| Code | Verdict | Meaning |
|---|---|---|
| 0 | PASS | All tests passed |
| 1 | FAIL | One or more tests failed |
These exit codes are deterministic and machine-parseable. No interpretation required.
- name: Run SystemEval Tests
env:
DEBUGGAI_API_KEY: ${{ secrets.DEBUGGAI_API_KEY }}
run: npx @debugg-ai/cli test
- name: Test PR via GitHub App
env:
DEBUGGAI_API_KEY: ${{ secrets.DEBUGGAI_API_KEY }}
run: npx @debugg-ai/cli test --pr ${{ github.event.pull_request.number }}
- name: Test PR Commits Sequentially
env:
DEBUGGAI_API_KEY: ${{ secrets.DEBUGGAI_API_KEY }}
run: npx @debugg-ai/cli test --pr-sequence
name: E2E Tests with SystemEval
on:
pull_request:
branches: [main]
push:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Full history for git analysis
- uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: Start dev server
run: npm run dev &
- name: Run SystemEval Tests
env:
DEBUGGAI_API_KEY: ${{ secrets.DEBUGGAI_API_KEY }}
run: npx @debugg-ai/cli test --wait-for-server --server-port 3000
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: systemeval-tests
path: tests/debugg-ai/
debugg-ai testGenerate and run E2E tests from git changes.
Authentication:
--api-key, -k - Your DebuggAI API key (or use DEBUGGAI_API_KEY env var)Git Analysis Options:
--last <number> - Analyze last N commits--since <date> - Analyze commits since date/time--commit <hash> - Test a specific commit--commit-range <range> - Test a commit rangePR Testing Options:
--pr <number> - PR number for GitHub App testing--pr-sequence - Test each commit in a PR individually--base-branch <branch> - Base branch for PR--head-branch <branch> - Head branch for PRLocal Development:
--wait-for-server - Wait for local dev server to start--server-port <port> - Port to wait for (default: 3000)--server-timeout <ms> - Timeout for server (default: 60000)--max-test-time <ms> - Maximum test wait time (default: 600000)Output Options:
--output-dir, -o - Where to save test files (default: tests/debugg-ai)--download-artifacts - Download test artifacts--verbose, -v - Enable verbose logging--dev - Enable development mode--no-color - Disable colored outputdebugg-ai statusCheck test suite status.
debugg-ai status --suite-id <uuid>
debugg-ai listList your test suites.
debugg-ai list --repo my-app --branch main --limit 10
import {
// Quick start function
runDebuggAITests,
// Core classes
CLIBackendClient,
GitAnalyzer,
E2EManager,
ServerManager,
// Configuration
DEFAULT_CONFIG,
ENV_VARS,
// Types
type CLIClientConfig,
type WorkingChange,
type CommitInfo,
type BranchInfo,
type WorkingChanges,
type GitAnalyzerOptions,
type PRCommitInfo,
type PRCommitSequence,
type E2EManagerOptions,
type E2EResult,
type PRSequenceResult,
type ServerConfig,
type ServerStatus,
type ServerManagerOptions,
type Chunk,
} from '@debugg-ai/cli';
The simplest way to run tests programmatically:
import { runDebuggAITests } from '@debugg-ai/cli';
const result = await runDebuggAITests({
// Required
apiKey: string;
// Optional - repository
repoPath?: string; // Default: process.cwd()
baseUrl?: string; // Default: 'https://api.debugg.ai'
// Optional - server
waitForServer?: boolean; // Wait for local server
serverPort?: number; // Default: 3000
// Optional - output
testOutputDir?: string; // Default: 'tests/debugg-ai'
downloadArtifacts?: boolean; // Download test files
maxTestWaitTime?: number; // Default: 600000 (10 min)
// Optional - PR testing
prSequence?: boolean; // Test each commit individually
baseBranch?: string; // PR base branch
headBranch?: string; // PR head branch
});
// Result type - deterministic verdict
interface RunDebuggAITestsResult {
success: boolean; // true = PASS, false = FAIL
suiteUuid?: string; // Non-fungible run identifier
testFiles?: string[]; // Generated test artifacts
error?: string; // Error message if failed
}
Analyze git repositories for changes:
import { GitAnalyzer } from '@debugg-ai/cli';
const analyzer = new GitAnalyzer({
repoPath: '/path/to/repo',
});
// Get working directory changes (uncommitted)
const workingChanges = await analyzer.getWorkingChanges();
// Get changes for a specific commit
const commitChanges = await analyzer.getCommitChanges('abc123');
// Analyze PR commit sequence
const prSequence = await analyzer.analyzePRCommitSequence({
baseBranch: 'main',
headBranch: 'feature-branch',
});
// Get enhanced context with branch info
const context = await analyzer.getEnhancedContext();
interface WorkingChange {
path: string;
status: 'added' | 'modified' | 'deleted' | 'renamed' | 'copied';
staged: boolean;
chunks?: Chunk[];
oldPath?: string;
}
interface CommitInfo {
hash: string;
message: string;
author: string;
date: string;
changes: WorkingChange[];
}
interface BranchInfo {
name: string;
tracking?: string;
ahead: number;
behind: number;
}
interface PRCommitSequence {
baseBranch: string;
headBranch: string;
commits: PRCommitInfo[];
totalCommits: number;
}
interface Chunk {
startLine: number;
endLine: number;
contents: string;
filePath?: string;
}
Full E2E test lifecycle management:
import { E2EManager } from '@debugg-ai/cli';
const manager = new E2EManager({
apiKey: 'your-api-key',
repoPath: process.cwd(),
baseUrl: 'https://api.debugg.ai',
testOutputDir: 'tests/debugg-ai',
serverPort: 3000,
serverTimeout: 60000,
maxTestWaitTime: 600000,
downloadArtifacts: true,
// Commit options
commit: 'abc123',
commitRange: 'main..HEAD',
since: '2 days ago',
last: 5,
// PR options
prSequence: true,
baseBranch: 'main',
headBranch: 'feature-branch',
});
// Wait for server to be ready
const serverReady = await manager.waitForServer(3000, 60000);
// Run tests - returns deterministic result
const result = await manager.runCommitTests();
// Cleanup resources
await manager.cleanup();
Manage local development servers:
import { ServerManager } from '@debugg-ai/cli';
const serverManager = new ServerManager({
port: 3000,
command: 'npm',
args: ['run', 'dev'],
cwd: process.cwd(),
env: { NODE_ENV: 'development' },
startupTimeout: 30000,
});
await serverManager.startServer();
const ready = await serverManager.waitForServerHealth(30000);
const status = serverManager.checkServerHealth();
await serverManager.stopServer();
Direct API communication:
import { CLIBackendClient } from '@debugg-ai/cli';
const client = new CLIBackendClient({
apiKey: 'your-api-key',
baseUrl: 'https://api.debugg.ai',
repoPath: process.cwd(),
timeout: 30000,
});
// Create a test suite from commit changes
const suite = await client.createCommitTestSuite({
changes: workingChanges.changes,
commitHash: 'abc123',
branchName: 'main',
});
// Check suite status
const status = await client.getCommitTestSuiteStatus(suite.uuid);
// List test suites
const suites = await client.listTestSuites({
repoName: 'my-app',
branchName: 'main',
limit: 20,
page: 1,
});
import { DEFAULT_CONFIG, ENV_VARS } from '@debugg-ai/cli';
// Default configuration values
DEFAULT_CONFIG.BASE_URL; // 'https://api.debugg.ai'
DEFAULT_CONFIG.TEST_OUTPUT_DIR; // 'tests/debugg-ai'
DEFAULT_CONFIG.SERVER_TIMEOUT; // 30000
DEFAULT_CONFIG.MAX_TEST_WAIT_TIME; // 600000
DEFAULT_CONFIG.POLL_INTERVAL; // 5000
DEFAULT_CONFIG.DEFAULT_SERVER_PORT; // 3000
// Environment variable names
ENV_VARS.API_KEY; // 'DEBUGGAI_API_KEY'
ENV_VARS.BASE_URL; // 'DEBUGGAI_BASE_URL'
ENV_VARS.GITHUB_SHA; // 'GITHUB_SHA'
ENV_VARS.GITHUB_REF_NAME; // 'GITHUB_REF_NAME'
ENV_VARS.GITHUB_HEAD_REF; // 'GITHUB_HEAD_REF'
ENV_VARS.NGROK_AUTH_TOKEN; // 'NGROK_AUTH_TOKEN'
| Feature | systemeval (Python) | @debugg-ai/cli (TypeScript) |
|---|---|---|
| Installation | pip install systemeval | npm install @debugg-ai/cli |
| CLI Command | systemeval test | debugg-ai test |
| Config File | systemeval.yaml | Environment variables |
| Exit Codes | 0/1/2 (PASS/FAIL/ERROR) | 0/1 (PASS/FAIL) |
| Adapters | pytest, jest, playwright | DebuggAI cloud E2E |
| Primary Use | Local test orchestration | CI/CD E2E generation |
Both implementations share the same philosophy: objective, deterministic, traceable test results.
| Variable | Description |
|---|---|
DEBUGGAI_API_KEY | Your API key (required) |
DEBUGGAI_BASE_URL | Custom API endpoint |
NGROK_AUTH_TOKEN | Ngrok authentication token |
GITHUB_SHA | Auto-detected in GitHub Actions |
GITHUB_REF_NAME | Auto-detected in GitHub Actions |
GITHUB_HEAD_REF | Auto-detected in GitHub Actions |
When --download-artifacts is enabled:
.spec.js).gif).json)Files are saved to tests/debugg-ai/ by default.
Authentication issues? Check your API key is set correctly.
Server not starting? Verify the port with curl http://localhost:3000.
No changes detected? Make sure you have git changes to analyze.
Tunnel issues? Check your NGROK_AUTH_TOKEN is set.
MIT
FAQs
CLI tool for running DebuggAI tests in CI/CD environments
We found that @debugg-ai/cli demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Axios compromise traced to social engineering, showing how attacks on maintainers can bypass controls and expose the broader software supply chain.
Security News
Node.js has paused its bug bounty program after funding ended, removing payouts for vulnerability reports but keeping its security process unchanged.
Security News
The Axios compromise shows how time-dependent dependency resolution makes exposure harder to detect and contain.