verdict-mcp

verdict

https://github.com/user-attachments/assets/62923912-98aa-4246-8bc8-4adf55ae3ff8

Structured, sandboxed verification feedback for coding agents.

An MCP server that replaces your agent's pytest shell-outs with something built for the agent inner loop: impact-selected tests, run in an isolated environment, returning compact typed verdicts instead of 40,000 tokens of raw runner output — with failure fingerprints that tell the agent whether a failure is its regression or was broken all along.

raw pytest dump:  ~40,000 tokens, unstructured, run un-sandboxed on your machine
verdict:              ~400 tokens, typed JSON, run in a rootless container, with memory

Why

The highest-frequency tool call in agentic coding is verification — and it's the least structured. Agents re-run whole suites when one module changed, burn context parsing ANSI-coded tracebacks, run arbitrary code directly on your machine, and routinely misdiagnose pre-existing breakage as their own regression (then "fix" code that wasn't broken). verdict fixes all four.

Tools

Tool	What it does
verify(scope?, base?)	Selects tests affected by your working-tree diff (static import graph via grimp), runs them via podman/docker with the worktree mounted read-only, returns typed failures with fingerprints and a preexisting flag
explain_failure(check_id)	Full traceback for one failure, on demand — bulk never rides in the summary
history(fingerprint)	First seen / last seen / times seen — regression vs. long-standing breakage
run_checks(["ruff","mypy"])	Lint and type checks, normalized into the same verdict schema

Every failure carries a fingerprint: a stable hash of the normalized failure signature (volatile tokens — addresses, tmp paths, ids, durations — collapsed). Same logical failure, same fingerprint, across runs and refactors. Fingerprints are what give verdict memory.

Quickstart

No install step needed — uvx fetches it on first use. (Or uv tool install verdict-mcp / pip install verdict-mcp for a permanent verdict-mcp command.)

Claude Code — .mcp.json in your project root:

{
  "mcpServers": {
    "verdict": {
      "command": "uvx",
      "args": ["verdict-mcp"],
      "env": { "VERDICT_PROJECT": "." }
    }
  }
}

Cursor — same shape in .cursor/mcp.json.

Optional verdict.toml in your repo root:

[project]
packages = ["your_package"]          # for impact selection (auto-guessed if omitted)

[runner]
image = "ghcr.io/you/yourproj-test"  # prebuilt env with your deps
setup_cmd = "pip install -e .[test]" # or install on the fly (runs with network; tests don't)
# prefer = "local"                   # escape hatch if you have no container runtime

[limits]
max_failures = 10

Try it without an agent:

cd examples/demo_project
VERDICT_PROJECT=. verdict-mcp   # then connect any MCP client, or use the MCP inspector

Sandbox posture (v0.1)

Checks run in an ephemeral container (podman preferred, docker fallback): worktree mounted read-only at /src, copied to a writable /work inside the container, --network=none for the check run. Your host environment is never mutated by a test run. If setup_cmd is configured, that step runs with network before the check; prefer a prebuilt image for a tighter posture. No container runtime → explicit prefer = "local" fallback runs checks against a temp copy of your worktree (still never in place). See SECURITY.md for the full threat model and known limitations.

Troubleshooting: if a verdict says container engine 'podman' could not start the check, run the suggested podman pull <image> by hand — the engine's own error is the answer. One known trap on macOS: a "credsStore": "gcloud" line in ~/.docker/config.json makes podman call the gcloud credential helper for every registry, including docker.io; an expired gcloud login then breaks all pulls. Fix with gcloud auth login or remove that line.

Honest limitations

• Impact selection uses the static import graph — approximate by design. Dynamic imports, fixture-by-name resolution, and data-driven tests can be missed; verify(scope="all") is always available and verdict says in selection_note whenever it falls back.
• Python/pytest only today, plus ruff/mypy. The adapter interface is small and documented — vitest and go test -json adapters are the most-wanted contributions (CONTRIBUTING.md).
• Flake detection and coverage-map-based selection are v0.2 (roadmap).

Roadmap

v0.2: coverage-based impact maps (precise selection), flake detection via fingerprint alternation, devcontainer.json support, result cache keyed on (tree hash, check, image digest). Later: vitest/jest, go test, cargo test adapters; per-repo verdict daemon mode.

License

Apache-2.0