@evalhub/cli

@evalhub/cli

EvalHub runner CLI. It runs evals locally and uploads result files as private drafts for review in the browser.

Install

npm install --global @evalhub/cli
evalhub --help

Requires Node.js >= 22.

Quick start: run and submit an eval

The submission flow is browser task → connect → run → validate → private draft → browser confirmation.

1. Create a submission task in the browser

Open the guide for the eval you want to run, for example:

https://fuckai.md/submit/result?eval=cold-jokes

Sign in, create a submission task, and keep the page open. The page shows a complete, one-time evalhub connect command. A task is bound to that platform origin, eval, user, and device.

2. Connect the CLI

Run the exact command shown on the page in your own terminal, then approve the connection in the browser:

evalhub connect PAIRING-CODE --platform-url https://fuckai.md

The CLI creates a PKCE verifier locally and exchanges the short-lived pairing request for a task credential. It never saves the pairing code or device code. The credential:

• is limited to result:draft:write;
• is bound to the exact task, eval, and platform origin;
• expires after at most seven days;
• is stored in ~/.evalhub/credentials.json (0700 directory, 0600 file).

Use evalhub status to list local task bindings. Remove a local binding with:

evalhub disconnect --task TASK-ID

Production platform URLs must be HTTPS. Plain HTTP is accepted only for localhost and 127.0.0.1.

3. Run the eval

The built-in runner calls an OpenAI-compatible /chat/completions endpoint configured in your terminal:

export EVALHUB_MODEL_BASE_URL="https://api.example.com/v1"
export EVALHUB_MODEL_API_KEY="sk-..."

evalhub run cold-jokes --model my-model-20260723

• The eval definition is fetched from EvalHub (https://fuckai.md by default). Set EVALHUB_PLATFORM_URL for another platform origin, or pass --local <dir> to use a local eval checkout.
• --model is the model ID reported by the provider (for example kimi-k3); it can also be set with EVALHUB_MODEL.
• Judge-scored evals additionally need EVALHUB_JUDGE_API_KEY.
• --adapter command runs a user-trusted local command instead of an HTTP API. The child receives only basic shell variables (PATH, locale, temp and terminal settings) plus a new empty temporary HOME/USERPROFILE that is deleted after the run. To pass an additional non-secret variable, list its name in EVALHUB_RUNNER_ENV_ALLOWLIST (comma-separated). Known credential variables are rejected, and common raw or encoded credential output is stopped before cache/result writes.
• The command adapter is not an OS sandbox. A malicious process running as the same user may still inspect user-readable files, process state, or the network, and can encode data in ways a content scanner cannot recognize. Run untrusted commands in a container or separate OS account with no host credentials or sensitive mounts.
• --dry-run reports planned tasks and model/judge calls without calling a model or writing a result.

On success, the CLI writes <slug>-result.json by default. Use --out <file> to choose another path.

Custom evals

Custom eval source is always pinned to a platform-provided repository commit. Download it without executing code, then inspect the printed directory, commit, README, dependencies, permissions, and environment-variable use:

evalhub fetch my-custom-eval

For an executable custom runner, rerun with the explicit execution acknowledgement only after review:

evalhub run my-custom-eval --allow-custom-code

An external_workflow eval cannot be run by evalhub run. Complete the README workflow on its required external infrastructure, keep the real submission manifest, and pass that file to the pinned packer:

evalhub pack my-external-eval \
  --input /path/to/actual-submission.json \
  --out my-external-eval-result.json

Example manifests document the shape only; they must not be used as the submission input.

4. Validate and upload a private draft

evalhub validate cold-jokes-result.json
evalhub submit cold-jokes-result.json --platform-url https://fuckai.md

evalhub submit selects an unexpired credential matching the exact platform origin and eval_id, validates the result against the current eval definition, and uploads the direct ResultFile as a private draft. If several tasks match, select one with --task <task-id>.

Success prints only:

• 私有草稿已上传;
• the browser review URL;
• the draft digest.

This does not publish the result.

5. Review and confirm in the browser

Open the Review URL returned by the CLI, verify the draft and digest, and explicitly confirm publication in the browser. A task credential cannot publish a result or perform author/admin actions.

If no matching task credential exists, evalhub submit prints the matching /submit/result?eval=... guide URL. It never falls back to a personal access token.

Validation options

evalhub validate <result.json> performs eval-aware validation without uploading. It loads the current eval definition from the platform by default. Use --local <dir> for a local eval checkout, or --schema-only only when an offline structural check is intentionally sufficient.

Legacy compatibility

evalhub login remains readable for older automation that already stores a PAT in ~/.evalhub/config.json, but the platform no longer creates PATs and they are not used by the normal evalhub submit flow. New submissions must use a browser-created task and evalhub connect. The public CLI accepts only results produced by eval runs.

Authoring evals

evalhub init my-new-eval

This creates evals/my-new-eval/ with eval.yaml, README.md, AUTHORS, sample-result.json, and tasks/ and assets/ directories, ready to fill in and open as a PR against the evals repository.

Development

This package lives in the EvalHub monorepo (packages/cli). Result files and eval definitions are validated with @evalhub/schemas.

pnpm --filter @evalhub/cli build
pnpm --filter @evalhub/cli test
pnpm --filter @evalhub/cli typecheck