ci-triage

ci-triage

Open-source CI failure triage for humans and agents. Parses CI logs, classifies failures, detects flaky tests, and outputs structured JSON — with an MCP server for coding agents.

v0.2 adds LLM root-cause analysis, a persistent SQLite flake database, and multi-CI support (GitHub, GitLab, CircleCI).

Why

When CI fails, everyone does the same dance: open the run, scroll logs, squint at errors. Coding agents have it worse — they can't scroll. ci-triage gives both humans and agents structured, queryable failure data.

Features

• 🔍 Smart log parsing — extracts errors, file:line references, stack traces
• 🏷️ Failure classification — 16 categories with severity levels and confidence scores
• 🧪 Flake detection — in-memory (single run) + SQLite persistent history across runs
• 🤖 LLM root-cause analysis — OpenAI Responses API, gated on OPENAI_API_KEY, graceful fallback
• 🌐 Multi-CI — GitHub Actions, GitLab CI, CircleCI, with auto-detection
• 📋 JUnit XML support — parses standard test result files
• 📊 Structured JSON output — agent-consumable schema with full failure context
• 🔧 MCP server — coding agents (Codex, Claude Code) can query failures programmatically
• ⚙️ GitHub Action — drop into any workflow with PR comments and artifacts

Quick Start

CLI

# Install globally
npm install -g ci-triage

# Or use without installing
npx ci-triage owner/repo

Basic usage

# Triage the most recent failed run
ci-triage owner/repo

# JSON output (for agents)
ci-triage owner/repo --json

# Triage a specific run
ci-triage owner/repo --run 12345

# Save markdown report
ci-triage owner/repo --md triage.md

v0.2 flags

# LLM root-cause analysis (requires OPENAI_API_KEY)
ci-triage owner/repo --llm

# Override LLM model (default: gpt-4.1-mini)
ci-triage owner/repo --llm --llm-model gpt-4o-mini

# Force a specific CI provider
ci-triage owner/repo --provider gitlab
ci-triage owner/repo --provider circleci

# Show flaky tests from persistent history
ci-triage flakes owner/repo

# Print version
ci-triage --version

Multi-CI provider setup

Provider	Token env var	Auto-detected by
GitHub Actions	gh auth login (uses gh CLI)	default
GitLab CI	GITLAB_TOKEN	.gitlab-ci.yml in cwd
CircleCI	CIRCLE_TOKEN	.circleci/ in cwd

Auto-detection runs at startup — pass --provider to override.

GitHub Action

- uses: clankamode/ci-triage@v1
  with:
    token: ${{ secrets.GITHUB_TOKEN }}
    flake-detect: true
    comment: true
    json-artifact: true

MCP Server (for coding agents)

Codex CLI (~/.codex/config.toml):

[mcp-servers.ci-triage]
command = "npx"
args = ["-y", "ci-triage", "--mcp"]

Claude Code:

claude mcp add ci-triage npx -y ci-triage --mcp

Tools exposed:

Tool	Description
triage_run	Full triage report for a CI run
list_failures	Recent failed runs summary
is_flaky	Flake history for a specific test
suggest_fix	Fix suggestions for failures

SQLite Flake Database

ci-triage persists run outcomes to ~/.ci-triage/flake.db. Over time, this builds a cross-session history of test pass/fail patterns, making flake detection much more accurate than single-run heuristics.

# After a few triage runs, list known flaky tests
ci-triage flakes owner/repo

# Output:
# Test Name                                               Fails  Passes   Ratio Last Seen
# ─────────────────────────────────────────────────────────────────────────────────────
# integration::auth::test_token_refresh                       3       7   30.0% 2026-02-25
# e2e::dashboard::renders_on_slow_network                     2       5   28.6% 2026-02-24

LLM Root-Cause Analysis

When OPENAI_API_KEY is set (or --llm is passed), ci-triage calls OpenAI Responses API to produce a human-readable root cause and fix suggestions:

OPENAI_API_KEY=sk-... ci-triage owner/repo --llm

# ── LLM Root-Cause Analysis ─────────────────────────────
# Model: gpt-4.1-mini (openai)
# Root Cause: The auth token refresh test fails intermittently due to a race
#             condition in the mock clock setup.
# Fix Suggestions:
#   • Use vi.useFakeTimers() with explicit tick advancement
#   • Add a 50ms buffer after token expiry before asserting refresh
#   • Consider extracting token timing into a testable utility
# Tokens: 1240 in / 187 out — est. $0.0008
# ─────────────────────────────────────────────────────────

If LLM is unavailable, ci-triage falls back to heuristic suggestions silently. The JSON output includes analysis.mode to distinguish.

Output Schema

{
  "version": "1.0",
  "repo": "owner/repo",
  "run_id": 12345,
  "run_url": "https://github.com/...",
  "commit": "abc1234",
  "branch": "feat/something",
  "status": "failed",
  "jobs": [{
    "name": "test",
    "status": "failed",
    "steps": [{
      "name": "Run tests",
      "failures": [{
        "category": "assertion_error",
        "severity": "medium",
        "error": "Expected 42 but received undefined",
        "file": "src/foo.test.ts",
        "line": 42,
        "flaky": { "is_flaky": true, "confidence": 0.92 },
        "suggested_fix": "Check null handling in Foo.validate()"
      }]
    }]
  }],
  "summary": {
    "total_failures": 1,
    "flaky_count": 1,
    "real_count": 0,
    "root_cause": "Flaky assertion in Foo.validate"
  },
  "analysis": {
    "mode": "llm",
    "provider": "openai",
    "model": "gpt-4.1-mini",
    "root_cause": "Race condition in mock clock setup",
    "fix_suggestions": ["Use vi.useFakeTimers() with explicit tick advancement"],
    "llm": {
      "usage": {
        "input_tokens": 1240,
        "output_tokens": 187,
        "estimated_cost_usd": 0.0008
      }
    }
  }
}

Failure Categories

Category	Severity	Description
oom	critical	Out of memory / heap exhaustion
port_conflict	high	Address already in use
permission_error	high	EACCES / permission denied
module_not_found	high	Missing dependency or import
assertion_error	medium	Test failure / assertion mismatch
docker_error	high	Container / image failure
network_error	medium	Connection refused / timeout
missing_env	high	Missing environment variable
missing_file	medium	File or artifact not found
rate_limited	medium	HTTP 429 / rate limiting
missing_lockfile	high	Lockfile not committed
dependency_security	high	Vulnerable dependency
timeout	medium	Step or job timeout
lint_error	low	Linter / formatter failure
type_error	high	TypeScript / compilation error
unknown	low	Unrecognized (manual triage)

Development

git clone https://github.com/clankamode/ci-triage
cd ci-triage
npm install
npm run build
npm test

License

MIT

Credits

Built by Clanka ⚡ — an autonomous engineer.