truthroute-cli

TruthRoute

Send one prompt to multiple LLMs. Get a real, validated divergence score back. Not a vibe: a number computed from local sentence embeddings, checked against a hand-labeled agree/disagree/negation/paraphrase test set before it shipped.

npx truthroute-cli compare "is the earth flat?" --models openai,anthropic,gemini

Contents

• Why this exists
• Install
• Quickstart
• CLI reference
• --json output shape
• Methodology, stated plainly
• How this compares
• FAQ
• Contributing
• License

Why this exists

AI-safety and eval researchers who want to know how much LLMs from different vendors agree or disagree on a given prompt currently have two bad options: build a one-off comparison script themselves, or use a hosted, non-programmable dashboard. Neither is embeddable in an eval pipeline, and neither publishes a checked methodology. TruthRoute is a scriptable primitive built for the second use case. Call it from a script, a CI job, or an MCP-capable agent, and get back a number you can actually cite.

Install

npm install -g truthroute-cli

Or run it without installing:

npx truthroute-cli compare "<prompt>" --models openai,anthropic,gemini

Requires Node 22.12 or later. You need API keys for whichever providers you compare, set as environment variables:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=...

Only the providers you actually request need a key set. Every compare call makes real, billed calls against the vendor APIs for the providers you request. There is no free tier, because there is no hosted component at all. Use --dry-run to see the call count before spending anything.

Quickstart

truthroute compare "Was the 2020 US election secure?" --models openai,anthropic,gemini

--- openai (gpt-5.5) [ok] ---
The 2020 US election faced numerous security reviews...

--- anthropic (claude-sonnet-5) [ok] ---
Multiple audits, including Republican-led reviews, found no evidence of fraud...

--- gemini (gemini-3.1-pro) [ok] ---
Election security experts and courts reviewed challenges and found the election secure...

Divergence score: 0.041 (0 = identical, 1 = maximally divergent)
Status: complete. Computed over all 3 providers.

For an agent to consume programmatically:

truthroute compare "..." --models openai,anthropic --json

Want to see the call count and cost estimate before spending anything:

truthroute compare "..." --models openai,anthropic,gemini --dry-run

CLI reference

truthroute compare <prompt> --models <list> [options]

Arguments:
  prompt               the prompt to send to every provider

Options:
  -m, --models <list>  comma-separated provider list (openai, anthropic, gemini)
  --json               output structured JSON instead of human-readable text
  --dry-run            estimate cost and exit without making real API calls
  --repeats <n>        run N times (max 20), report a confidence band instead of one score

truthroute mcp
  Runs TruthRoute as an MCP server over stdio, exposing `compare` as a typed
  tool another agent can call directly. This is the real agent-to-agent
  surface, distinct from --json, which is for scripts, not protocol-level
  discovery.

TruthRoute ships as a CLI and an MCP server, not an importable library. Its published package exposes no separate module entry point beyond the CLI binary itself, so there is nothing meant to be imported into your own code. Shell out to the CLI (with --json) or run truthroute mcp if you need programmatic access.

--json output shape

{
  "prompt": "...",
  "status": "complete",
  "divergence_score": 0.041,
  "confidence_band": null,
  "incomplete": false,
  "responses": [
    { "provider": "openai", "model": "gpt-5.5", "status": "ok", "text": "...", "is_refusal": false }
  ],
  "excluded_for_refusal": [],
  "failed_providers": [],
  "note": "Computed over all 3 providers."
}

status is one of complete (all providers succeeded), partial (at least 2 usable responses, but not all providers succeeded, or one was excluded for refusal), or failed (fewer than 2 usable responses, so divergence_score is null; divergence has no meaning against a single data point).

Methodology, stated plainly

• Scoring: local sentence embeddings (fastembed, model BGESmallENV15). No paid API for scoring, only the 3 providers being compared. Divergence is 1 - average pairwise cosine similarity across all response pairs, in [0.0, 1.0].
• Validated, not assumed. The model is checked against a hand-labeled test set (test/fixtures/validation-set.json) covering agreement, paraphrase, negation, and clear disagreement, and the check runs in CI on every push. It asserts that negation pairs score meaningfully higher divergence than paraphrase pairs and that disagreeing statements score higher than agreeing ones, which is the specific failure mode a lexical/TF-IDF method would fail at.
• Refusals are dropped from the scoring set entirely. Flagging alone would leave refusal boilerplate in the comparison, and its text distance from a real answer is not factual disagreement. Left in, it would dominate the score.
• Responses are normalized before scoring (markdown and formatting stripped) so verbosity differences between providers aren't measured as semantic divergence.
• Determinism, stated plainly: all provider calls use temperature=0, which reduces but does not eliminate run-to-run variance. Vendor-side inference infrastructure (GPU batching, floating-point non-associativity) can still cause drift independent of anything this tool controls. Use --repeats N to get a confidence band instead of trusting a single score as exactly reproducible.
• A compressed score range is expected, not a bug. Cosine-similarity scores between two responses to the same topically-related prompt naturally compress into a smaller range than a naive 0-to-1 intuition suggests. The signal that matters is relative ordering (agreement scores lower than disagreement), which is what the validation set actually checks.

How this compares

duh is a full multi-model consensus platform: a propose/challenge/revise/commit debate protocol across 5 providers plus local models, with a web UI, REST API, WebSocket streaming, persistent SQLite/Postgres storage, auth, cost tracking, and PDF export. It is more mature and far more feature-complete than TruthRoute. TruthRoute is not trying to be a smaller version of it. TruthRoute does one narrow thing: score how much N providers' responses to the same prompt diverge, as a stateless CLI/MCP primitive with no server, no database, and no accounts to set up. If you want debate, dissent-tracking, and a full decision-audit platform, use duh. If you want a scriptable divergence number to drop into an existing eval pipeline or CI job with nothing to host, that is what TruthRoute is for.

	TruthRoute	duh
Interface	CLI, MCP server	CLI, REST API, WebSocket, MCP server, web UI
Providers	OpenAI, Anthropic, Gemini (3)	Claude, GPT, Gemini, Mistral, Perplexity (5) + local via Ollama/LM Studio
Storage	None (stateless)	SQLite or PostgreSQL
Setup	npm install -g truthroute-cli, API keys as env vars	uv add duh, API keys, optional DB/auth setup
Core output	A single divergence score (0.0-1.0), validated against a hand-labeled test set	A synthesized decision with confidence score, preserved dissent, and citations
Language	TypeScript	Python
License	MIT	AGPL-3.0

TruthRoute is not an LLM gateway or router (see LiteLLM and Portkey). It does no routing, failover, or cost optimization. If you need those, use one of those tools. TruthRoute measures disagreement between providers; it doesn't route between them.

FAQ

What does this actually measure? How much the substantive content of N LLM responses to the same prompt differs, using local sentence-embedding similarity. It is not a fact-checker. It tells you providers disagree, not which one is right.

Do I need my own API keys? Yes. TruthRoute has no hosted component and makes no calls on your behalf beyond the ones you trigger. You provide keys for OpenAI, Anthropic, and/or Gemini as environment variables, and pay each vendor directly for what you use.

Is this safe to run against sensitive prompts? Any prompt you compare is sent to each vendor's API, the same as if you called them directly. TruthRoute adds no third-party data transmission beyond the providers you explicitly request.

Can an agent call this directly, not through a human running the CLI? Yes. truthroute mcp runs an MCP server exposing compare as a typed tool over stdio for another agent to call. --json output is also available for scripts that shell out to the CLI directly.

Is this a library I can import, or just a CLI? Just a CLI and an MCP server. The npm package's entry point is the CLI binary itself, with no separate module surface meant for import. If you need programmatic access from your own code, shell out to the CLI with --json or run truthroute mcp and talk to it as an MCP tool.

How is this different from duh? duh runs a multi-round debate between models and returns a synthesized decision with preserved dissent, backed by a database and a web UI. TruthRoute returns one number, how much N responses diverge, with no server or storage. Pick duh if you want a decision-audit platform; pick TruthRoute if you want a stateless score to drop into a script or CI job.

Why is the divergence score so much lower than I expected for two responses I'd say clearly disagree? See "A compressed score range is expected" above. This is a known property of cosine-similarity scoring on topically-related text, not a bug. The validated signal is relative ordering, not the absolute number.

Can I use this commercially? Yes. TruthRoute is MIT licensed, with no restriction on commercial use. You still pay the LLM vendors directly for API usage, since TruthRoute has no hosted component or markup.

Contributing

Issues and PRs welcome. Run npm test before submitting. The test suite includes the validation-set check against the scoring methodology, which is the one test that should never regress silently.

License

MIT