@evalhub/schemas

@evalhub/schemas

Shared zod schemas and validators for EvalHub:

• EvalDefSchema / StoredEvalDefSchema — eval definitions (eval.yaml)
• ResultFileSchema — locally executed runs and author-imported upstream publications
• validateResultForEval / validateParticipantForEval — cross-checks between a result and its eval definition
• buildEvalCommandPlan — resolved run/submit command plans for custom runners

Used by @evalhub/cli and the EvalHub platform.

An eval may expose verified first-party project links without overloading its description:

references:
  homepage: https://example.org/benchmark
  paper: https://arxiv.org/abs/2607.25886
  repository: https://github.com/example/benchmark

Reference URLs must use HTTPS and cannot contain URL credentials. repository means the upstream project repository; EvalHub derives its own eval source link from the official eval repository.

Custom runners have two modes:

• Omit custom_mode (legacy-compatible) or set custom_mode: executable when evalhub run --allow-custom-code can execute the pinned command directly.
• Set custom_mode: external_workflow when the participant must first run on external infrastructure. Its command_template.argv must contain one standalone {input} token for the real submission manifest and one standalone {output} token for the generated result.

runner: custom
custom_mode: external_workflow
command_template:
  argv: [node, evals/example/pack-to-result.mjs, "{input}", --out, "{output}"]
  output: example-result.json

The CLI resolves these placeholders as separate process arguments; they are not shell interpolation points.

Install

npm install @evalhub/schemas

Usage

import { ResultFileSchema, validateResultForEval } from "@evalhub/schemas";

const result = ResultFileSchema.parse(JSON.parse(raw));
const checked = validateResultForEval(evalDef, result);
if (!checked.success) {
  console.error(checked.error.issues);
}

result.submission is a provenance union:

• kind: "run" (or omitted for backward compatibility) requires runner_version and run_date.
• kind: "upstream_author_publication" records an upstream HTTPS source, retrieval date, importer version, and SHA-256 snapshot digest. Eval-aware validation requires a non-null numeric score. It is valid for both local- and author-scored evals because scored_by describes recognition authority, not whether a first-party publication exists.

Eval definitions separate two score requirements:

• protocol_revision is a positive, monotonically increasing score-protocol revision (default 1 for legacy definitions). Increase it when tasks, execution, primary scoring, or tiebreak comparability changes. Do not increase it for copy, README, references, or baseline-data maintenance.
• score_policy: required rejects every score: null submission. Use author_fill only when the eval author must receive an artifact before they can assign the primary score.
• baseline_policy: required requires at least one reviewed numeric result in published-results/*.json. Repository sync imports those envelopes using the eval author's identity and binds them to the current eval commit.

Omitted score_policy values remain backward compatible: scored_by: author resolves to author_fill, while scored_by: local resolves to required. New evals should declare the policy explicitly.

results[] is the rankable primary-results table. Each row's score is its primary value; an eval may also declare a raw_metric.tiebreak_value rule for equal scores. The resulting eval rank derives global leaderboard points. results[].supplementary_views may add structured metric_table or line_chart displays for component scores, trends, and alternate analyses; these views never become score or tiebreak inputs.

Upstream publications may identify the base model in participant.model and a separate product/agent harness in participant.harness. A harness version is optional when the upstream source did not publish one; when present, harness_version requires harness. Upstream participants cannot include config. Publications may include supplementary_views, but cannot include run-only usage, task_results, or showcases. This keeps published component scores visibly separate from locally executed per-task evidence.

Example result-row fragment:

{
  "participant": { "model": "example-model" },
  "score": 32.59,
  "supplementary_views": [
    {
      "type": "metric_table",
      "title": "Six published component scores",
      "columns": ["SWE-V", "SWE-M", "SWE-Pro", "GPQA", "AIME", "TB2"],
      "rows": [{ "cells": [33, 15, 9, 65, 53.33, 20.22] }],
      "note": "Auxiliary display only; does not affect ranking."
    }
  ]
}

Only score (plus an eval-declared tiebreak for equal scores) ranks this row. Values inside supplementary_views are display-only.

Every accepted result is bound by the platform to the commit that introduced the current protocol_revision. Supplying a different eval_commit is rejected as stale, and leaderboards only select results bound to that protocol commit. The latest source commit is tracked separately, so copy-only maintenance does not clear the current leaderboard; increasing protocol_revision deliberately starts a new comparable score cohort.

Development

This package lives in the EvalHub monorepo (packages/schemas).

pnpm --filter @evalhub/schemas build      # tsc -> dist/
pnpm --filter @evalhub/schemas test       # vitest
pnpm --filter @evalhub/schemas typecheck