evaldog

evaldog

Run LLM / prompt evals locally — deterministic grading, zero LLM tokens.

Built for CI gates and AI agents that need a fast pass/fail on prompt/RAG outputs without burning context window.

npx evaldog run cases.csv
npx evaldog run cases.csv --min 80     # exit 1 if score < 80 (CI gate)
npx evaldog run cases.csv --json       # machine-readable (for agents)

File formats

CSV — name,output,expected,assert

name,output,expected,assert
Password reset,Click the reset link.,reset link,contains
JSON shape,"{""ok"":true}",,is-json

JSON / YAML

cases:
  - name: greeting
    output: "Sure! Happy to help."
    assert:
      - { type: not-empty }

Assertions: contains · icontains · equals · regex · is-json · not-empty (default contains when an expected value is present, else not-empty).

Why agents like it

An agent can grade 200 outputs with one shell call and read back a single number

• exit code — instead of streaming every case through the model.

evaldog run outputs.csv --json --min 90 || echo "regression!"

Hosted dashboard + scheduled drift alerts: https://evaldog.com

MIT © The Testing Academy