@machinegrade/validate

machinegrade validate

Validate AI-generated artifacts against a contract before you act on them:

• json_schema — validate artifact against a JSON Schema (ajv, all errors collected).
• openapi_response — validate a response body against the response schema for a given path + method + status in an OpenAPI spec.
• sql — check a SQL string for syntax errors in a given dialect.

Every check returns a verdict, not an error: {valid, errors, latency_ms}, HTTP 200 whether the artifact is valid or not. Only genuinely wrong requests (bad key, unsupported type, malformed body, over your limit) get typed HTTP errors.

Built on Hono — one codebase, runs locally on Node today and is written to be Cloudflare Workers-compatible for deploy later (see "Deploy" below).

Why

Agents that generate JSON, API responses, or SQL need a fast, cheap, machine-checkable pass/fail before they ship the result — cheaper than a full LLM-as-judge call, and deterministic.

Run it locally

npm install
npm run dev
# machinegrade validate listening on http://localhost:8787

3 runnable examples

1. curl

# Get an API key
curl -s -X POST http://localhost:8787/keys \
  -H 'content-type: application/json' \
  -d '{"email": "[e-mail]"}'
# => {"key":"sk_..."}

# Validate a JSON artifact against a JSON Schema
curl -s -X POST http://localhost:8787/v1/validate \
  -H 'content-type: application/json' \
  -H 'X-Api-Key: sk_...' \
  -d '{
    "type": "json_schema",
    "artifact": {"name": "Ada", "age": 30},
    "contract": {
      "schema": {
        "type": "object",
        "required": ["name", "age"],
        "properties": {"name": {"type": "string"}, "age": {"type": "number"}}
      }
    }
  }'
# => {"valid":true,"errors":[],"latency_ms":1}

2. Python (requests)

import requests

base = "http://localhost:8787"

key = requests.post(f"{base}/keys", json={"email": "[e-mail]"}).json()["key"]

resp = requests.post(
    f"{base}/v1/validate",
    headers={"X-Api-Key": key},
    json={
        "type": "sql",
        "artifact": "SELECT id, name FROM users WHERE id = 1",
        "contract": {"dialect": "mysql"},
    },
)
print(resp.status_code, resp.headers.get("X-Calls-Remaining"), resp.json())

3. MCP config snippet

mcp/server.ts exposes a single tool, validate, that forwards to POST /v1/validate. Point an MCP-compatible client at it:

{
  "mcpServers": {
    "machinegrade-validate": {
      "command": "npx",
      "args": ["tsx", "mcp/server.ts"],
      "cwd": "/path/to/validate",
      "env": {
        "SANDBOX_URL": "http://localhost:8787",
        "SANDBOX_API_KEY": "sk_..."
      }
    }
  }
}

API

See public/openapi.yaml for the full contract, or /v1/manifest for a machine-readable summary (types, limits, pricing, error codes) once the service is running. /llms.txt is a short pointer for LLM agents.

Endpoint	In	Out
POST /keys	{email}	{key}
POST /v1/validate	header X-Api-Key, body {type, artifact, contract?}	verdict, header X-Calls-Remaining
GET /v1/manifest	—	capability manifest
GET /stats	header X-Admin-Token	funnel: keys_issued, active_callers, repeat_callers_7d, limit_hits, paid_requests
POST /v1/paid-request	header X-Api-Key	records interest in paid access
GET /openapi.yaml, GET /llms.txt	—	static docs

Pricing

• Free tier: 500 calls/month per key, 60 calls/minute rate limit.
• Paid tier: EUR 0.002/call beyond the free tier — opens soon. Request paid access via POST /v1/paid-request (requires X-Api-Key); you'll be notified when it's live.

Errors

Every error is typed JSON — {code, message, hint, docs_url} — never a free-form string:

Code	HTTP status	When
INVALID_KEY	401	X-Api-Key missing or unknown
LIMIT_EXCEEDED	402	Free-tier monthly limit (500 calls) exceeded
UNSUPPORTED_TYPE	400	type is not json_schema, openapi_response, or sql
MALFORMED_INPUT	400	Request body doesn't match the documented shape
RATE_LIMITED	429	More than 60 calls/minute for a key

A verdict ({valid, errors, latency_ms}) is never an error — an invalid artifact is a normal, expected outcome and returns HTTP 200.

Storage

src/storage.ts defines a Storage interface with two implementations:

• MemoryStorage — full in-memory implementation, used for npm run dev and the test suite.
• D1Storage — real Cloudflare D1 binding, backed by schema.sql (keys, events tables). Used in production; the Workers entry point in src/index.ts builds it from the DB binding on first request.

Apply schema.sql to a new D1 database with:

wrangler d1 execute machinegrade-validate-db --file=schema.sql          # local
wrangler d1 execute machinegrade-validate-db --file=schema.sql --remote # production

Testing

npm test          # vitest run, in-process via app.request(), MemoryStorage
npm run typecheck # tsc --noEmit

Tests cover: key issuance, happy + fail cases for each validator, typed 401/400/402/429 errors, the metering limits (both injectable in tests so they don't require looping hundreds of real requests), and /stats funnel counts.

Deploy

This template runs on Cloudflare Workers (Hono + D1 + Workers Static Assets). To deploy to a fresh Cloudflare account:

wrangler d1 create machinegrade-validate-db   # copy the returned database_id into wrangler.toml
wrangler d1 execute machinegrade-validate-db --file=schema.sql --remote
wrangler secret put ADMIN_TOKEN
wrangler deploy

Then bind a custom domain (e.g. api.machinegrade.dev) to the Worker via the Cloudflare dashboard or wrangler. CI can deploy on push to main once CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID repo secrets are set and the deploy job in .github/workflows/ci.yml is uncommented.

Two things worth knowing about the Workers port:

• GET /openapi.yaml and GET /llms.txt are served by the ASSETS binding ([assets] in wrangler.toml, pointing at public/) — Cloudflare serves them directly, without invoking the Worker. The routes in src/index.ts are a fallback for local Node dev/tests, where there's no ASSETS binding.
• The json_schema and openapi_response validators use @cfworker/json-schema, not ajv: ajv compiles schemas via new Function(...), which the Workers runtime disallows, and schemas here arrive dynamically per request (from the caller), so they can't be precompiled at build time either.

Kill criteria

This is a demand-test sandbox for one experiment (see ../experiments/EXP-001-output-validation.md). It's built to be disposable: if the experiment doesn't show demand, delete this directory without ceremony.