numguard

numguard

_{MCP registry identity — mcp-name: io.github.ipezygj/numguard}

The verification layer for the agent economy — an agent-callable primitive that checks a number before it gets asserted, and hands back a signed receipt proving it was checked.

Agents now produce an explosion of numbers: eval scores, A/B results, "the agent improved 12%", benchmark rankings, backtest Sharpes. The scarce resource isn't the number — it's trust in the number. numguard is the tool an agent calls mid-task to ask "does this survive a second look?", and to attach a portable, tamper-evident receipt so the answer travels with the claim.

Built on evalgate for the shared eval statistics; adds the pieces agents specifically need — a Deflated Sharpe Ratio for backtests, judge calibration, signed receipts, and metering an agent can actually pay (prepaid credits + x402 pay-per-call). Exposed as an MCP server, so any agent can call it.

New here? — How to verify a backtest is real (Deflated Sharpe in Python): the practical guide to catching an overfit or leaking backtest, with runnable code. See it work — proof gallery: 8 real numbers run through the real checks, 3 survive and 5 are flagged, each with a receipt you can verify offline. Don't trust it — verify it. Wire it into an agent in one line — INTEGRATE.md: the local reflex, an MCP config, and LangChain / CrewAI tool wrappers.

The tools

MCP tool	What an agent asks it
verify_backtest	Is this strategy's Sharpe real, or the luckiest of the many I tried? (Deflated Sharpe Ratio)
verify_backtest_series	Run the full integrity battery on my actual returns — look-ahead, autocorrelation, regime, tail, overfitting.
verify_subset_win	Does "we lead on subset X" survive correcting for how many subsets I tested?
verify_model_gap	Is the gap between these two models bigger than the test set can resolve?
verify_judge_bias	Is my judge's preference real, or just longer / first / same-family?
calibrate_judge	Is the LLM judge I trust actually calibrated against ground truth?
audit_leaderboard	Is #1 on this leaderboard statistically real? (rank confidence intervals)
triage (start here)	I don't know which check I need — here's what I'm about to do or assert, route me. (front door across numguard + agent-guard + evalgate, free)
verify_execution	Don't trust my reported Sharpe — RE-DERIVE it from my positions on committed price data, and catch a number those decisions don't produce.
reconcile_backtest	Did my backtest's claimed Sharpe survive contact with LIVE returns? (HELD / DECAYED / BROKEN)
open_commitment / report_returns	Hold my strategy accountable over time — stream live returns, tell me when the edge breaks. (O(1)/obs)
open_precommitment / report_precommit	Prove my live claim wasn't cherry-picked after the fact — pre-register it BEFORE the outcome; report on a hash-chained, tamper-evident timeline anyone can audit free (verify_chain).
issue_receipt / commitment_receipt	Give me a signed, portable proof this number / track record was checked.
verify_receipt	Was the number this other agent handed me actually checked, and by whom? (free, issuer-agnostic)
scan_for_receipts	A peer just sent me a message — find and verify any receipt inside it before I act. (free — the receiver half of the loop)
receipt_spec / why / pricing / balance	the open receipt standard · what numguard does that nothing else does · prices · balance

What sets it apart (why): computing the number yourself, or a lesser checker, stops at "is it significant?" numguard also holds it accountable to live reality over time, signs a portable tamper-evident proof, and lets anyone verify any proof for free — the trust layer, not just a calculator.

For agent traders: the Deflated Sharpe Ratio

The number that kills a backtest is the same one that kills a benchmark score: you tried many, and you reported the best. In finance the rigorous correction is the Deflated Sharpe Ratio (Bailey & López de Prado) — given how many variants you tested, what Sharpe would the luckiest zero-skill strategy have shown, and do you beat it after adjusting for sample length and non-normal returns?

from numguard import deflated_sharpe

deflated_sharpe(sr=0.12, T=250, n_trials=100)
# SR=0.120 over T=250, 100 trials tested; deflation bar=0.160; DSR=0.263
# -> does NOT survive deflation.  (PSR-vs-0=0.970 — it LOOKS significant on a single test.)

deflated_sharpe(sr=0.15, T=1000, n_trials=1)
# DSR=1.000 -> SURVIVES. A real edge over a long sample.

The contrast is the whole point: a single-test probability of 0.97 ("significant!") collapses to a deflated 0.26 ("noise") once you account for the 100 strategies that were tried. An agent optimizing over strategies should call this before it trusts — or publishes — a backtest.

The full integrity battery — what a Deflated Sharpe still misses

DSR catches best-of-N. It does not catch same-bar look-ahead, autocorrelation inflating the Sharpe, regime dependence, tail fantasy, or one-lucky-epoch fragility. verify_backtest_series runs the whole battery on the actual returns series and returns a risk level (none/medium/high/critical) plus the checks that flagged:

check	catches
leakage	same-bar look-ahead (position "predicts" the bar it's in) — critical
pbo	overfitting beyond n_trials (Prob. of Backtest Overfitting) — critical
hac_sharpe	autocorrelation / stale marks inflating the Sharpe (Newey–West)
regime_stability	cherry-picked window (per-block Sharpe + CUSUM break)
bootstrap_stability	edge lives in one epoch (block-bootstrap Sharpe CI)
drawdown	tail/smoothing fantasy (Calmar / CVaR / expected-vs-realized max-DD)
permutation, conditional_hetero, cost_capacity, bh_fdr	order structure, vol clustering, fill realism, multiple testing

The tell (python examples/catch_a_fake_backtest.py): a look-ahead strategy shows an annualised Sharpe of +20 and a Deflated Sharpe that survives — yet the battery flags it critical on leakage (same-bar corr 0.79 vs next-bar 0.05). The DSR waves the fiction through; the battery does not.

verify_backtest_series(api_key="…", returns=[...], positions=[...], asset_returns=[...])
# {"risk": "critical", "survives": false, "flags": ["leakage", ...], "checks": {...}}

Signed receipts (the part that compounds)

from numguard import verify_claim, issue_receipt, verify_receipt, keypair
priv, pub = keypair()
result  = verify_claim("backtest", sr=0.12, T=250, n_trials=100)
receipt = issue_receipt(result, priv, pub)     # Ed25519-signed
verify_receipt(receipt)                          # True — anyone can verify with the public key alone

Attach the receipt to your output. A downstream agent (or human) can confirm — without your keys — that the claim and its verdict weren't altered and that numguard issued them. As receipts circulate, "a number without a receipt" starts to read like "a number nobody checked."

Buying is easy for an agent

Two rails, both built so an agent can decide and pay in-loop, no human clicking:

• Prepaid credits + API key — a human tops up once; the agent spends per call. Generous free tier (25 calls/key) so the agent feels the value first, then a machine-readable price list. Insufficient balance returns a structured payment_required, not an error.
• x402 pay-per-call — the agent hits a tool, gets an HTTP-402 with a machine-readable price + pay-to address, pays USDC from its wallet, retries with proof, gets the result. The protocol layer is here; settlement is pluggable (inject a facilitator/RPC verifier for production).

from numguard import x402
x402.require_payment("verify_backtest", price_usd=0.03, pay_to="0x…")
# -> {"status": 402, "accepts": [{"scheme":"exact","network":"base","asset":"USDC", ...}]}

Run the MCP server

pip install git+https://github.com/ipezygj/numguard
python -m numguard.mcp_server        # stdio MCP server; point your agent/host at it

Then an agent calls e.g. verify_backtest(api_key="…", sr=0.12, T=250, n_trials=100) and gets a verdict it can quote and a receipt it can attach.

Deploy it (hosted, paid, discoverable)

1. Host the paid HTTP API (x402 per-call):

docker build -t numguard . && docker run -p 8080:8080 \
  -e NUMGUARD_PAYTO=0xYOURWALLET \
  -e NUMGUARD_FACILITATOR_URL=https://your-x402-facilitator \
  numguard

Or one-click on Render: New → Blueprint → this repo (render.yaml included); set NUMGUARD_PAYTO + NUMGUARD_FACILITATOR_URL in the dashboard. With NUMGUARD_PAYTO unset the API runs free (dev mode) so you can test before wiring a wallet. Endpoints: POST /verify_backtest, /verify_model_gap, … ; GET /pricing.

The x402 flow, end to end: the agent POSTs → gets 402 with an accepts block (price, payTo, network) → signs a USDC payment → retries with an X-PAYMENT header → numguard verifies + settles it through the facilitator to your wallet → returns the result. Settlement is the real x402 /verify + /settle handshake (numguard.x402.facilitator_verifier) — facilitator-agnostic: point NUMGUARD_FACILITATOR_URL at any x402 facilitator. Options:

• Testnet (free, no account): https://x402.org/facilitator with NUMGUARD_NETWORK=base-sepolia — test the whole flow with test-USDC first.
• Mainnet, self-sovereign: self-host x402-rs (open-source, no third party) and point at your own URL.
• Mainnet, hosted (non-Coinbase): thirdweb or PayAI facilitators (Base) — set NUMGUARD_FACILITATOR_AUTH if the facilitator needs a key.

2. Serve the MCP server over HTTP (for remote MCP hosts): uvicorn numguard.mcp_server:app (or NUMGUARD_TRANSPORT=streamable-http python -m numguard.mcp_server).

3. Get discovered: server.json (official MCP registry) and smithery.yaml (Smithery) ship in the repo; connect the repo at those registries so agents can find the server. GitHub topics: mcp, mcp-server, x402.

Design notes

• Statistics are shared with evalgate (zero-dependency); numguard adds the backtest, receipt, metering, and MCP layers on top — it does not re-implement the core checks.
• Pure-math numerics where possible; cryptography only for Ed25519 receipts (HMAC fallback without it).
• Every verdict is derived from a computed statistic, never asserted — the same discipline as the book behind it, Measured, Not Believed (leanpub.com/measurednotbelieved).

MIT.