benchkeeper

benchkeeper

Installs, updates and undoes the BenchKeeper agent setup for Claude Code, Codex, Gemini CLI and GitHub Copilot, globally and per project, from a CLI or a local web UI. What the setup contains: the setup guide.

Quickstart

Node.js 22 and at least one agent, Claude Code or Codex (full list). Choose the App to review each change in the browser. Choose the CLI in a terminal-only session or in a script.

App

npx benchkeeper@6.26.0

Opens the browser on the Dashboard with the doctor findings (the read-only checks of this machine). Select Fix everything, review the changes and press Apply; nothing is written before.

CLI

npx benchkeeper@6.26.0 plan

Lists each change and writes nothing.

npx benchkeeper@6.26.0 apply

Asks per change. --yes applies the default selection without questions.

After either path, restart the agents. npx benchkeeper@6.26.0 history lists what changed, npx benchkeeper@6.26.0 undo <run> reverses a run.

Lite level, the plugin alone. Full setup description. What the app writes and what it never does.

Setup

Node.js 22 or newer, and at least one agent. Linux and macOS are tested; Windows runs only a CI smoke test (start, plan, apply with a ledger run) and is untested by the owner.

• From npm: npx benchkeeper@6.26.0 <command>. A clone of the branch stable runs the latest release (each release moves it); to run a change that is not on npm yet, clone the default branch (--branch main).
• From a clone: git clone --depth 1 --branch stable https://github.com/ESchuderer/BenchKeeper.git && npm ci --prefix BenchKeeper/app --omit=dev, then node BenchKeeper/app/bin/benchkeeper.mjs <command>. The web UI needs a build first: npm ci --prefix BenchKeeper/app && npm run build --prefix BenchKeeper/app.

Runtime dependencies, pinned: jsonc-parser 3.3.1 (JSON settings with comments), smol-toml 1.9.1 (Codex TOML), yaml 2.9.1 (workflow files for projects audit). The web UI (React, Mantine) is built into dist/ and ships in the package.

Commands

benchkeeper help lists every option; benchkeeper help <command> [verb] or benchkeeper <command> [verb] --help prints only that command's lines. An unknown command prints one line on stderr (exit 2), and an old name from before the noun-first rename names its new command.

Command	Does
ui [--no-open] [--port N]	the web UI (default command), see Web UI
plan, apply [--yes] [--all]	global setup for the installed agents (--agents a,b, --pack matt-pocock): each change (file changes as a diff, commands as their command line), then a question per change; --yes applies the default selection without a question; --all alone lists the removals and asks once whether to apply all changes, including removals; --yes --all applies every change including removals without a question. Neither removes a plugin BenchKeeper did not install. Only the BenchKeeper plugin is pinned (the branch stable, which each release moves, at least at this app's version); Ponytail and Matt Pocock's skills follow their source. The agent's own update, run by hand (third-party marketplaces have auto-update off by default), moves an installed plugin along the ref its marketplace was added at; a marketplace at another ref stays there until you remove and add it again (Update). A registered host account whose BenchKeeper plugin is older than this app's version gets update steps, a newer one is kept. Plugin state is read from ~/.claude/plugins/ and ~/.codex/config.toml, the agent CLIs only when those cannot be read. A plugin whose marketplace was added at another ref than the pin is a step by hand, with the commands: moving the marketplace removes it, which deletes the plugin's data. An installed pack plugin whose pack no registered project uses is offered for removal, not preselected (uninstalling deletes the plugin's data); one BenchKeeper did not install is kept unless you answer its own question (--yes and --all skip it), with the command that adds its pack to a project and the one that brings it back. apply takes --json (see Machine-readable output)
apply --level lite|standard|full	the install level; default: the level found, full on a machine without BenchKeeper. lite and standard plan the BenchKeeper plugin alone and nothing else on the machine; undo uninstalls it. standard adds the local block that projects apply writes per repository. doctor names the level it finds
history show <run>	one run: each change with its status and the diff of each file it wrote (settings values masked), a recorded session's files between its snapshots (the first 50)
history, undo <run> [--step s] [--unit u], restore <name>	recorded runs, undo of a run, step or unit (an installed skill, pack, plugin, rules or agent files: history show --json names each change's unit), restore of a setup.sh backup folder. A command is undone with its recorded reverse command (an install with its uninstall); a command without one, such as removing a plugin you had, is reported as manual. Undo puts a file back only while it is as the run (or the recorded session) left it and its backup is intact; otherwise it fails for that change and names the file. Each change is journaled before it starts, so a change cut off by a crash shows in history and can be undone. undo takes --json (see Machine-readable output)
projects plan|apply [dir]	project setup in dir or --project <dir> (both naming different folders is refused): --agents, --env container|host-agent|sandbox|host, --sandbox, --no-guards (no deny rules or guard hook; --guards turns them on again), --machine on|off|none (Claude Code: the machine account's variables in the project's .claude/settings.local.json, or with off your own git identity, gh's default folder, no GIT_CONFIG pairs and no guard address over the values of machine enable --global; none takes the setting out; kept in the state file), --no-workflow-load (no rule line that loads the workflow skill before each task; --workflow-load puts it back), --pack, --skill, --tier, --share or --local, --agent-file <path> (agent files of a source, see skills sources; --agent-file none takes them out), --caller lint-js|lint-python|sonar (writes that repo-standards caller workflow as a file in the checkout; one that differs from yours is offered, not preselected; no GitHub setting is ever changed). Local (the default): the block goes into CLAUDE.local.md and AGENTS.override.md (its block holds the text of AGENTS.md too, rebuilt on each apply; your lines outside it stay), the sandbox into .claude/settings.local.json, and every file written is listed in a marked block of .git/info/exclude (the repository's, shared by its worktrees); --share writes the block into AGENTS.md with a CLAUDE.md import and keeps only CLAUDE.local.md and AGENTS.override.md in the exclude block. At a repository root, a marked block of .worktreeinclude lists the local files a new worktree copies (Claude Code --worktree, Orca): local, the files written (.benchkeeper/state.json, not the folder) and the list itself, all git-ignored; --share, committed, only CLAUDE.local.md and AGENTS.override.md. Your lines outside the block stay. skills disable <name> moves a skill to .benchkeeper/disabled/, skills enable <name> moves it back. Answers and each community skill's commit are kept in .benchkeeper/state.json; a community skill is added again from the default branch of its source only when you ask for it (skills update, or Update in the Catalog), and the file then records the commit the install got. Undo of a skills CLI command puts back the skill folders and skills-lock.json as they were before it, local edits included. A changed answer cleans up: sandbox off removes sandbox.enabled and BenchKeeper's excluded commands, a dropped agent loses the CLAUDE.md import, the Gemini CLI context setting and skill folders no chosen agent reads. A warning names hand-written AGENTS.md text on where commands run. projects apply takes --json (see Machine-readable output)
sessions list [--run id]	one line per orchestrated run (Orca workers, matched by the Orca run: line and the task id of the worker's first prompt) with sessions, cost (API-equivalent; Codex estimated from its token counts and OpenAI's API prices) and tokens, from the same scan as sessions board; --run adds a line per task of that run and exits 2 when no session has that run
sessions cache show|clear|set [--keep-days N|none] [--yes]	the summaries the sessions scan keeps after an agent deletes a transcript: show counts them with the retention; set --keep-days (1 to 3650, none: for good) changes the retention, the next scan drops older ones; clear removes every summary, those of deleted transcripts for good. Both show the change and ask. The command cannot see a scan the web UI runs, which writes the old summaries back: clear while none runs
sessions record [--project dir] -- <agent> [args]	runs an agent session in a Git repository and records the files it changed (tracked and untracked, not ignored ones) and its Claude Code or Codex transcript when found, so undo restores them; exits with the agent's exit code (127 when it does not start)
rules show <file> [--agent a] [--project dir]	the text of one file rules list shows
skills show <name>	the SKILL.md of a library skill BenchKeeper ships or of your own; for any other skill its source
automations [list], automations show <id>	the automation templates of both tiers and which are installed; one template with its fields, parameters and prompt
automations install <id> [--param name=value..] [--project dir] [--yes], automations update|remove <id> [--yes]	install creates the automation on the runner (Orca, through orca automations create), disabled, in the project folder (default: the current one), and records template id, version, parameter values and the runner's id in automations.json of the config folder; undo removes it. update shows the diff, then edits the runner's automation back to the template (its enabled state stays); undo edits it back. remove deletes it on the runner with its run history and drops the record: undo restores the record only, so status reports it removed and install creates it again. Without Orca the three refuse and change nothing
automations export <id> [--param name=value..] [--project dir]	prints a sh script that runs the template without the runner: it unpacks the helpers into a temporary folder, runs the precheck in the project folder (default: the current one) and starts claude -p or codex exec only when the precheck exits 0. Its header has the crontab line of the template's schedule. Refuses a template without a cron schedule, another agent or workspace new-worktree. Nothing is written (Without Orca)
automations diff <id>, automations status	the installed automation as the runner holds it against the template now; per installed automation current, outdated (the template changed), modified (changed on the runner: the fields), removed (gone from the runner) or unknown (runner or template unreadable), exit 1 when one is not current. Nothing is written
containers reconcile [--dry-run]	stops and removes (podman rm -f, or Docker) each dev container whose workspace folder (its devcontainer.local_folder label) no longer exists while the folder above it does, so a worktree removed without its container is caught; volumes stay. A container without that label, whose folder exists or cannot be read (permission denied), or whose label is not an absolute path inside the home folder (another user's folder, a Windows path, a folder on a remote engine's host) is never touched, and a missing parent folder (an unmounted drive) keeps the container. --dry-run lists them. Meant for cron or a timer; exit 1 when a removal fails. Linux only: other systems skip it and say so. doctor (level full) reports, by count only, these orphaned containers, stopped ones still present, ones whose folder could not be checked and unlabelled ones, and warns when the user key quota (/proc/key-users against kernel.keys.maxkeys) is at 80 percent or more, since each running container holds keys and new ones fail at the limit
webhook set|remove|status, webhook send --state finished|failed|needs-person [--run id] [--automation id] [--task id]	the webhook a run calls when it finishes or needs a person, off until set. set reads the URL (https, or http to this machine, no user or password in it) from the first line of stdin and an optional bearer token from the second, and keeps both in the OS credential store under app.webhook, never in a file; remove deletes them after a question; status shows whether one is set and the last failed send, never the URL or token. send posts one JSON body {"event":"run","state",["run"],["automation"],["task"],"time"} with ids of fixed shapes only (no objective text; another shape exits 2), from a detached process with a 5 second limit, no redirect and no retry, and returns at once; without a webhook it does nothing. A failure appends the time and an HTTP status or error code, without the URL, to .cache/benchkeeper/webhook.log in the home folder. Nothing calls send on its own: a run that is killed sends nothing
machine setup [--repos owner/name,..] [--owner name] [--no-signing] [--no-browser] [--yes] [--project dir], machine status	your GitHub machine account for agent sessions (setup). You create the account yourself: setup opens GitHub's sign-up, suggests the login <your login>-agent (your login from your own gh api user), and names the profile note and two-factor authentication. Then gh auth login --hostname github.com --git-protocol https --web --scopes write:ssh_signing_key signs it in with GH_CONFIG_DIR set to machine/gh in the config folder, with no terminal on stdin so gh asks nothing (it neither offers itself as git's credential helper nor adds the workflow scope; GIT_CONFIG_GLOBAL points at a file of the machine folder as well); the device page opens and gh prints the code. The token stays in gh's store, BenchKeeper never reads or prints it. A sign-in as your own account (the same id) is refused and signed out. Access: with your gh, PUT /repos/{owner}/{repo}/collaborators/<machine> with permission=push (never admin or maintain) for --repos, every repository of --owner (listed, archived ones left out, after a question), else the folder's origin; then the machine's gh accepts the invitations; a refused repository (an organization's policy) is named with the fix and exits 1. Signing (default on, --no-signing skips): an ed25519 key in machine/signing_key (0600) uploaded with POST /user/ssh_signing_keys. machine.json keeps both accounts by numeric id and their logins for display, the gh path and the signing choice, never a token. status reads both logins again (a rename is written back; the commit address follows), shows the sign-in, the address and the key
machine enable|disable --global [--yes], machine shell [--project dir], machine remove [--yes]	enable --global writes the machine variables into the env of Claude Code's user settings and into [shell_environment_policy.set] of Codex's config.toml, where ~/.claude or ~/.codex exists, as a plan with a diff (undo restores the files); disable --global takes out exactly these names. The variables: GIT_AUTHOR_* and GIT_COMMITTER_* (<login>, <id>+<login>@users.noreply.github.com); GIT_CONFIG_COUNT pairs that reset the credential helpers of https://github.com and set !GH_CONFIG_DIR=<machine folder> <gh> auth git-credential, send pushes to an ssh address of github.com over https (pushInsteadOf), and with signing gpg.format=ssh, user.signingkey=<key> and commit.gpgsign=true; GH_CONFIG_DIR (every gh call acts as the machine account); BENCHKEEPER_MACHINE_EMAIL for the guard hook. A project: projects apply --machine on|off|none. shell starts your $SHELL with your environment minus GH_TOKEN, GITHUB_TOKEN and outer GIT_CONFIG_* pairs plus these variables; nothing goes into a shell start file. remove asks, takes the variables out of the agent settings (a run), signs the machine account out of the machine folder, deletes the folder with the key and config, and prints the pages to delete the signing key or the account (signed in as the machine account)
packs warnings <id>	the pairs of the pack's skills whose descriptions overlap, so an agent may pick the wrong one
projects advise [dir] [--agent claude|codex]	the env, packs and skills an agent suggests for a registered project, each with a reason: one read-only session on your login (default: the first installed of Claude Code, Codex), refused in CI
skills sources [list]	agent-file sources: each file a pick (--agent-file engineering/engineering-frontend-developer.md), fetched with curl from raw.githubusercontent.com at a pinned commit into the folder of each chosen agent the source installs for: .claude/agents/<name>.md for Claude Code, .codex/agents/<name>.toml for Codex (converted to its name, description and developer_instructions fields) and .github/agents/<name>.agent.md for GitHub Copilot CLI; a file goes to the backup when its pick or its agent is dropped. Ids and paths come from the catalog, never from a project's state file.
packs list|show|save|export|import|remove	library packs and own packs; show <id> adds the registered projects that use it; a pack file names library skills only; remove refuses an own pack a registered project's setup still names and lists those projects
packs enable|disable <id>.. [--project dir | --global] [--yes]	switches packs without a new setup: a plan on top of the saved answers, with a diff and an undo like any run. --project (default: the current folder) changes the project's packs, and disabling removes exactly the skills no other enabled pack and no single pick still needs; --global installs or removes the plugin of a pack that has one. Enabling and then disabling the same pack returns the project's files to what they were. packs enable and disable take --json (see Machine-readable output)
packs scope <id> [global|projects] [--yes]	a pack's install scope. global installs it once on the machine for every project; projects installs it in each registered project whose setup selects it (Claude Code plugins at local scope, on this machine only). Defaults: the plugin packs global, the skill packs projects. Without a scope it prints the current one and the scopes each agent offers, with the reason where an agent offers one only (Codex installs plugins for the user only; its other scopes are unverified). With a scope it plans the move: out of one place, into the other, and the scope recorded, with a diff and an undo like any run. A pack at global is never offered for removal because no project selects it
skills [list], skills update|reset|remove|adopt|add|enable|disable [name..] [--project dir] [--yes]	skills of one project: list gives each skill's state (current, outdated, modified, removed, disabled), the commit installed and, with --check, the commit the source's default branch is at (one git ls-remote per source, per repository, so a repository with many skills counts as moved when any file changed), and the skill folders no BenchKeeper setup installed (untracked, also in the agents' global folders; no update or apply touches them). update adds the named skills again from the source's default branch and brings removed ones back (no name: checks the sources, then updates every outdated one); reset replaces a skill edited here (the plan shows the diff and the edit goes to the run's backup; a plan without reset leaves an edited skill alone); remove takes a single pick out of the project (a skill a pack brings leaves with the pack); adopt makes an untracked library skill a single pick of the project and keeps its folder; add makes a library skill the project lacks a single pick (a name is required); disable moves a skill to .benchkeeper/disabled/ and enable moves it back (a name is required). Own skills are compared with the library's files, community skills by the commit and by the hash of their folder recorded when the install ran. skills update takes --json (see Machine-readable output)
skills new <name> [--file path | --description text], skills edit <name> --file path, skills delete <name>	your own skills, one folder each in ~/.config/benchkeeper/skills/<name>/ (SKILL.md and any other files). The name follows the Agent Skills rules (1 to 64 lowercase letters, digits and single hyphens, repeated as name in the front matter, with a description of 1 to 1024 characters) and must not be a library skill's. When a newer library adds a skill of the same name, the library's wins: doctor and skills list --library name the clash, and a project that picked yours keeps its files until the skill is reset. A project or own pack picks one like a library skill, and the setup copies its folder into the agents' skill folders; an edit reaches a project on its next apply. new starts from --file or a SKILL.md with the name and --description; edit replaces the SKILL.md; delete is refused while a registered project, own pack or preset names the skill. projects plan|apply --skill refuses a name that is neither a library skill nor yours (exit 2); a stored pick that no longer resolves stays picked, and the plan warns. Each is a plan with a diff, undone in History. packs export refuses an own pack that names your skills, naming them: a pack file carries library skills only. Each takes --yes and --json
projects list|add|scan|remove|rename|relocate|unshare|confirm-git, projects update [dir..] [--yes] [--dry-run] [--unattended]	registered projects (add refuses another worktree of a registered repository unless --worktree; rename <dir> [name] [--notes text] sets the display name and notes in projects.json; relocate <old> <new> moves the entry, its kept answers and its history to a moved folder; remove only unregisters, remove <dir>.. --setup [--yes] lists every file, then takes BenchKeeper's setup out and keeps the project registered: a local setup as one run in the checkout (benchkeeper undo <run>), a shared one as a commit on the local branch chore/remove-agent-setup); projects update takes registered project folders to update only those (--dry-run and --baseline follow the list; an unregistered folder is refused, exit 2); --unattended skips and lists a project whose repository git config that runs code or sends git elsewhere (hooks, core.hooksPath, core.sshCommand, credential helper, push URL, URL rewrites, filters, signing and the like) differs from the list confirmed with projects confirm-git <dir>.., which shows that list (values redacted, hooks by name) and stores keys and value hashes in projects.json after a yes typed in a terminal (no --yes; exit 2 when stdin is not a terminal); --unattended does not imply --yes. Update all applies a local setup in the checkout, and for a shared one builds branch chore/agent-setup-<version> per project in a worktree and commits; it pushes nothing, and each result names the git -C <dir> push -u origin <branch> command while the branch holds commits to push. A repository that still tracks .benchkeeper/state.json while the setup is local gets a commit that removes the committed setup (committedSetup: the AGENTS.md block, an import-only CLAUDE.md, BenchKeeper's sandbox keys, the recorded and locked skills, skills-lock.json, .benchkeeper/) on branch chore/agent-setup-local; projects unshare does the same for the projects named or all. The checkout stays as it is until the merge is pulled, and the state file is kept in projects.json until Update all has written the local setup again. An unmerged branch from an earlier run gets the new commit on top and may hold only commits Update all made (recorded by id in the ledger); a project whose apply or commit fails keeps its worktree under .cache/benchkeeper/worktrees/ in the home folder until you remove it. projects add, remove, rename and relocate take --json (see Machine-readable output)
projects create <dir> --preset <id> [--yes] [--all], presets [list|save <id> --project dir|remove <id>]	a preset is the setup answers of a project (agents, environment, packs, single skills, tier and the rest a profile keeps), saved under a name in presets/ of the config folder; a pack is a set of skills switched on in a project, a preset starts a new project with its packs. presets save copies the answers of a project with a setup and never replaces a preset of the same id; remove moves the file to the backup. projects create refuses a folder that exists, a missing parent folder, a folder inside a registered project and an unknown preset; otherwise it shows the setup plan with a diff, applies the chosen changes and registers the folder (no git init: projects apply in the repository later adds the git exclude list). A check then recommends, each with its reason, skills to turn off (also installed in a global skill folder, or a description that overlaps another of the project's skills) and packs to add (on in at least half of the other registered projects that share an agent, two projects at least), as a second plan with no change selected: --yes takes none of them, --all all
projects status [dir]	app version, install level, the agents with their BenchKeeper plugin, and per registered project (or only dir) the setup version, origin's setup, the checkout, branch, clean state and the pending changes; nothing is written
skills list --library	the library's skills with the number of registered projects that have each
skills cli	checks the registry signatures of the skills CLI release a skill install runs (the newest release at least 7 days old, never below the app's minimum) and prints one line, the environment (npm_config_before holds its dependencies to releases at least 7 days old) and npx -y skills@npm:skills@<version>, to start every skills CLI command with; exit 1 when the check fails
inventory, dashboard	the global setup items with their state; the Dashboard: item states of the machine and of each registered project, packs per agent, catalog
evals results	the stored eval results: miss and false-trigger rates per target, agent, agent version and model
projects audit [dir] [--only workflows,tests,code,findings] [--json]	where the project's CI and tests are bloated or in bad shape, as a Markdown report (--json: findings); reads the checkout and runs no project command, tool or test, writes nothing. dir defaults to the current folder. Checks: workflows (a push trigger that also runs every pull request branch, no cancel of superseded runs, tests or build on prose-only changes, an install without a cache, the same build in two jobs, a matrix of more than 8 combinations, actions not pinned to a commit and broad permissions, both marked as security, not bloat); tests (a coverage floor more than 10 points below the measured value, from package.json scripts for node --test, c8 and nyc, the c8 and nyc objects in package.json, .c8rc.json, .nycrc, .nycrc.json or the jest threshold, against coverage/coverage-summary.json or coverage/lcov.info; identical token runs in test files; snapshots or fixtures totaling over 1 MiB per kind); code (identical token runs of at least 100 tokens and six lines in the project's own source, comments and import lines left out; files over the max-lines of .eslintrc.json, else 1000 lines); findings (a lint step that continue-on-error, || true or || exit 0 keeps from failing the run). A check whose need is missing (no workflow, no coverage report, no floor) is listed as skipped with the one-line way to enable it, never as a failure, and no project is told to install a tool. A finding never suggests lowering a threshold, removing a check or adding a suppression. Scans at most 5000 files (also of the workflows folder) of 512 KiB each and 500000 tokens per duplication check (code and tests each), skips a file or folder it cannot read, reports the 20 longest clones of each duplication check
sessions board [--no-open]	the sessions board: writes board.html in the benchkeeper config folder (readable only by you) and opens it. One row per Claude Code and Codex session from the transcripts in .claude/projects/ and .codex/sessions/ below your home folder (Codex subagent rollouts count toward their parent): project, branch, tier and change type from the Tier: lines, the pattern (loop, group, Orca worker or single, see the README; Claude Code transcripts only, Codex rollouts give group), setup labels from the skills installed in the session, the BenchKeeper version of the last global apply before it (from ledger/), skills used, the last test command's result, the pull request links the transcript records (not their state), time, tokens, and the cost Claude Code records (API-equivalent; for Codex an estimate from its token counts and OpenAI's standard API prices of the date the footnote names, marked estimated, models without a published price left out). The page has a project rail (newest activity first, with session count, sparkline and last tier; a chip row at phone width), tiles for sessions, agent time, cost, tokens and tests-pass rate, each with a sparkline and its change against the previous period of the same length (none for all time), a per-day chart stacked by project with a metric switch (cost, agent time, tokens, sessions) and legend chips that hide a series, comparison bars by tier, pattern and setup with the sample size on every bar (a bar under 20 sessions, the threshold of insights, is hatched, marked in words and left out of the highest and lowest line), model mix by week, histograms of agent time and turns with the median marked, a 12-week activity calendar with active days and the longest streak, and for one chosen project its branches and their pull requests, and a table of cost and tokens per orchestrated run and task over all periods. Every chart has a table of values behind a button. The page makes no request and holds no message text. Parsed files are cached by size, modification time and a parser version in sessions.json in the benchkeeper cache folder (readable only by you). The cache keeps the summary of a session (project, tier, pattern, setup, tokens, cost, tests, pull requests, times, title, and the run and task ids of an Orca worker; never message text) after the agent deletes its transcript, so history older than cleanupPeriodDays comes from this cache and starts when the app first scanned; a newer app converts the cache, and a cache file it cannot read (damaged, or written by a newer app) is moved aside as sessions.json.<time> with a note, never overwritten; Preferences sets how many days summaries stay (default: for good) and clears the cache. The first scan reads every transcript (34 s for 8.9 GB on the test machine), later ones only changed files. Nothing leaves the machine
insights --json [--project dir] [--agents a,b] [--narrative --agent claude|codex [--yes]]	findings about this machine's setup from five signals, nothing written except the cached narrative of --narrative --yes: installed Claude Code skills (global folders and enabled plugins) that no session of the last 90 days used, with the tokens their descriptions cost per turn (needs 20 sessions in that window, no finding below it); friction, a project (by its path, shown by its folder name) with at least 20 sessions in that window, not counting Orca workers and research-tier sessions, of which some ended on a failed last test run (the rate and the session ids, no message text; Claude Code and Codex); the doctor's findings, as hygiene; versions behind (the doctor's outdated items), as freshness; skill proposals per project (--project, else every registered project with a setup), as skills: turn off the skills none of the project's last 20 or more sessions in the window used (skills disable; a skill that another skill the project sees names or links is never proposed), add the skills of library packs the repository matches as the setup suggests them (skills add), update the ones behind the library's copy (skills update); each fix is that command with --project, which shows the plan with its diff and undo before it writes; skills behind their source need the network and are in skills list --check. Only with --json; the Insights view shows the same findings. --narrative adds a summary with a suggested order, written by one session of the agent named with --agent on your login (no tools, an empty folder; it covers the findings the Insights view shows, the triage applied, and narrative.covers lists their ids, at most 25, errors first). Without --yes it prints the exact input to stderr, runs nothing and exits 2, unless a text for the same findings is cached: that text is printed (cached true) and no session starts. With --yes it prints the input to stderr, runs the session and prints the text; it does not run in CI. The input is only the findings' fields (id, signal, severity, the finding's message, counts, window and sample size, proposal text; a project by its folder name; a path under the home folder, the config folder or a registered project is cut to its last segment, also when a folder name has a space; any other path is cut to its last segment up to its first space): no session messages, prompt, file content or skill description
doctor [dir] [--project dir] [--baseline auto|oss|private-advisory] [--baseline-ref ref]	reports without changing anything (--baseline adds, per GitHub repository, the drift of merge methods, rulesets, required checks, Actions allowlist, private vulnerability reporting, Dependabot, code scanning, secret scanning and the repo-standards caller workflows against a profile; read-only gh api GET requests, advisory for private repositories, a note instead of an error without gh, a login or the network; the profile is read from repo-standards at the pinned commit, with a bundled copy as fallback; applying a profile is repo-standards's scripts/apply.mjs, with a dry run, a snapshot and a restore; projects update --baseline <profile> prints the same report per registered project and changes no setting): variables that turn agent features off (DISABLE_TELEMETRY, DO_NOT_TRACK, CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC, DISABLE_GROWTHBOOK, CLAUDE_CODE_DISABLE_ADVISOR_TOOL; name and file, never the value), rules and settings that differ from the setup (rules you edited are a warning, and the plan's optional "keep your edits" change stops it until the file or the setup version changes), items outside it, agent settings that cancel each other or do nothing (the advisor with a switch that turns flag fetching off, Remote Control with a switch that makes it unavailable, an update channel with the nonessential-traffic switch, a top-level effortLevel that differs from the levels under modelSettings, includeCoAuthoredBy next to attribution, auto or bypassPermissions in a project or local file, autoMode in a project or local file, a key the settings reference marks deprecated or removed, cleanupPeriodDays below 30 against the Sessions board, CLAUDE_CODE_SUBPROCESS_ENV_SCRUB not on, no Read deny rule for each of the .ssh, .aws and .config/gh folders, the Claude Code sandbox on macOS, Linux and WSL2 (off, on without sandbox.failIfUnavailable, sandbox.allowUnsandboxedCommands not false, bubblewrap or socat missing on Linux and WSL2; native Windows is reported as unsupported without a warning; an empty sandbox.network.allowedDomains is advice, never written), a Codex sandbox_mode of danger-full-access, switches that count as set with 0 or false, OpenTelemetry variables in a project env, a Codex project notify, default_permissions with sandbox_mode), Claude Code settings files it would skip, Codex trust entries of folders that are gone, a missing container engine; in a project: summary or layout sections in AGENTS.md, overlapping skill descriptions, broken skill links (the skill, its project and the missing target), a local setup the repository still has committed; instruction files (global and project): an @import in CLAUDE.md, CLAUDE.local.md or a rules file whose target does not exist (file and line, imports followed up to four hops), a file over 200 lines (the memory documentation's target, with its link), skills of different scopes (project, global, enabled Claude Code plugins) whose descriptions overlap, with file and line of both; allow rules (permissions.allow of the user, project and local files and Codex .rules files), reported as a warning for a bare Bash (or Bash(*)), a wildcard program, Bash(<shell or interpreter> *) or the same with -c or -e (bash, sh, python, node, npx and similar; Bash(python -m pytest *) is not reported), and as information for a bare Edit, Write or WebFetch (or Tool(*)) and a * before the rest of the command (Bash(git * main)), each with scope, file, rule and how many registered repositories carry it; nothing is changed and no rule is proposed. Automations: an installed one that drifted from its template or the runner, a template without a precheck, two installed ones with the same schedule in one project checkout (Automations). Exit 1 on an error, see Machine-readable output
doctor level line	the install level it finds (lite: the plugin, standard: a project setup too, full: the global rules too) and what the next level adds; below full only the plugin is expected
settings list [--agent claude|codex] [--advanced] [--project dir], settings get <key>, settings set <key> <value>|--unset|--bundle <id> [--layer user|project|local]	the agents' own settings (Claude Code settings.json, Codex config.toml): the app's list per agent with default, value and the file it comes from, notes on keys that cancel each other, and with --advanced every key of the pinned schemas and the settings reference plus keys the app does not know (shown and kept). A change is a plan with a diff and an undo like any other; the value is checked first (strict JSON for Claude Code, a line edit that keeps comments and key order for Codex), managed settings are never written, an environment switch is set (value set) or removed, never 0 or false, and values of names that look like secrets are never printed. projects.<path> with --unset --agent codex removes a trust entry. settings set takes --json (see Machine-readable output)
rules list [--agent claude|codex] [--project dir], rules disable|enable <file> [--agent a] [--project dir]	the instruction files Claude Code and Codex load, with owner (BenchKeeper, project, yours), size, load state and the context budget. disable leaves a file out without deleting it: for Claude Code through claudeMdExcludes in the user settings (with --project: in that project's .claude/settings.local.json, so one project only); for a global Codex file by moving it into <config>/disabled-rules/, which enable reverses byte for byte. Both are runs, undo restores them. rules disable and enable take --json (see Machine-readable output)
rules check [--agent a] [--project dir] [--limit tokens]	checks the files rules list shows, without a model: a file over the size limit (default 10000 tokens, BenchKeeper's own), auto memory beyond the 200 lines or 25 KB Claude Code loads, an @import, link or npm run script that does not exist, the same line in two files one agent loads, a path-scoped rule whose globs match no file, an empty or unreadable file, a rule file no agent reads. Exit 1 on an error finding. Doctor lists the same findings; the Rules view shows them above the files
extensions list [--agent claude|codex] [--project dir]	the MCP servers, hooks and subagents Claude Code and Codex load, plus the commands and output styles of Claude Code (none for Codex), read from files only (no agent starts): kind, scope, owner (BenchKeeper, plugin, project, managed, yours), the file, and a runs-code mark for hooks with a command, stdio MCP servers and commands that run a shell line. Environment values and headers never appear, a URL shows scheme and host only, a command line is masked. Read-only
profile export [file] [--origins], profile import <file> [--clone-into dir]	agents, packs and the pack scopes you chose, own packs and registered projects with their answers (guards and agent files included) as one file (no file contents, environment values or tokens); import asks once before it saves the own packs and registers the projects found at the same paths, then asks per change. --origins adds each project's origin address (https or ssh of a named host, a login in it dropped, listed before the file is written). Import lists a missing project that has an origin, then clones it into dir (asked for when missing) after its own question, with no hooks and no submodules; nothing is cloned under --yes. Registered and set up like any project afterwards; undo removes the folder only while the clone is untouched (the setup run first)
knowledge show [dir] [--json], knowledge clear [dir] [--yes]	the knowledge base the maintaining-repos skill keeps per project in .benchkeeper/knowledge/ (git ignores it; one JSON file per kind: verdicts, test audit results, architecture map, tool results, open questions). show prints the entries of each file; a file that does not parse, is a link or is over 1 MiB is listed as ignored, and a linked .benchkeeper is not followed. clear asks, then removes that folder (also an empty one) and nothing else (state.json and the rest of .benchkeeper/ stay); the base is rebuildable, so there is no undo. dir defaults to the current folder
accounts [list], accounts add [name] [--agent claude|codex] [--container --mount path] [--label text] [--dir path], accounts use|remove <name>, accounts status [name] [--project dir], accounts track <name> [--off] [--project dir] [--yes], accounts limits [name] [--five-hour N] [--seven-day N] [--warn-five-hour N] [--warn-seven-day N] [--pool name|none] [--auto-pick on|off] [--when-none stop|wait] [--continue-below on|off] [--rotate-claude off|handover] [--reset], accounts handover <name> [--project dir] [--yes], accounts login <name> [--token] [--email address] [--no-browser] [--project dir]	several logins per agent, set up through a guided sign-in. An account is a folder of its own on the host (CLAUDE_CONFIG_DIR or CODEX_HOME, by default .local/share/benchkeeper/accounts/<name> below your home folder, mode 0700, never inside the config folder) or a named volume mounted in the project's dev container (--container --mount <path>: the folder in the container where the volume is mounted, one its user can write, below its home folder; there is no default because the owner of a new volume depends on the image). The registry, accounts.json in the config folder, holds name, agent, folder or volume, label, default and, for a kept token, its creation date: no credential. add, use and remove are plans with a diff, a run in history and an undo (remove also deletes a stored token, which undo does not restore, and leaves the account's folder and sign-in alone). login starts claude auth login (--email pre-fills the address) or codex login --device-auth, takes the sign-in link from its output and opens it in a private window of Firefox (--private-window), Chrome or Chromium (--incognito) or Edge (--inprivate), so the sign-in page asks which account to use; with no such browser, on a machine without a display, for a link of an unknown host or with --no-browser it prints the link (Codex also its one-time code) for you to open. When the browser shows a code, paste it at the prompt and it goes to the agent. --token: Claude Code setup-token (a subscription's one-year token that makes model requests only), kept in the OS credential store (libsecret through secret-tool, macOS Keychain through security; other systems and a missing service refuse, there is no file fallback), recorded in accounts.json as a date, never shown; Codex: the access token piped on stdin goes to codex login --with-access-token in the account's folder and is kept nowhere else. A container account runs the sign-in inside the container (--project, default the current folder): devcontainer up starts it with every container account's volume as a mount, which applies when a container is created (the command that recreates it is printed when the volume is not mounted), and devcontainer exec --remote-env CLAUDE_CONFIG_DIR=<mount> runs the agent; the volume survives a rebuild. list and status run claude auth status or codex login status with the account's environment (a container account in the --project dir's running container) and reports a login that is gone and a kept token older than 11 months (error once a year old) with the command that fixes it; doctor reports the same under check accounts. Usage of a signed-in Codex host account: status, doctor and run start codex app-server with the account's CODEX_HOME and read account/rateLimits/read once (windows of 300 and 10080 minutes, usedPercent and resetsAt; nothing else of the answer is kept); thresholds and findings as below, no failure records, no auto-pick. Usage (Claude Code): track copies one script into <account folder>/benchkeeper/ and sets it as the account's statusLine and as a StopFailure hook (matcher: the 12 error types the hooks page documents; a three-type entry an earlier version wrote is replaced when track runs again) in the account's settings.json, next to the hooks already there; an existing statusLine of another kind makes the plan an error. The script keeps usage.json (per window used and resetsAt, the read time) and failures.jsonl (time and error type, the last 20), mode 0600, nothing else of the input. A host account's change is a plan with a diff, a run in history and an undo; a dev container account's files are written through devcontainer exec (--off reverses, no run). status, doctor and the Accounts view read the files (a container's through devcontainer exec cat): headroom is 100 minus used_percentage; a window whose reset time has passed is unknown, never full; below the minimum a warning, below the notice level (default twice the minimum) an info, and, among the last 20 records, the latest failure of each of the 12 types within 24 hours (a type whose record was pushed out by later failures gives no finding; a warning for authentication_failed, oauth_org_not_allowed, account_on_hold, billing_error, model_not_found and cloud_credential_error, an info for the others). limits sets the minimum headroom and notice level per window (defaults 8 and 3 percent, notice 16 and 6) for all accounts or one, the pool of an account, and the launch settings; they live in accounts.json (settings, and limits and pool per account) and are changed as plans with undo. --rotate-claude handover (off by default) turns on rotation by handover for Claude Code; Codex has none (--rotate-codex handover is refused, a stored one counts as off). handover then checks the account with the same thresholds: with a window below its minimum, both windows known and another account of its pool at or above its minimum, it prints the current account, the target and the handover text. On yes (--yes or --continue-below on: no question), it writes .benchkeeper/handover.md in the worktree and starts the agent as the target through the run launcher, with the text as the first prompt. The text has task and goal, done, open, worktree and branch, uncommitted file names and the next step, from git and the worker's .benchkeeper/handoff.md. It has no transcript text, and token shapes are masked. No credential moves between accounts. Setting handover and each handover print the provider terms warning with its sources (Claude Code: Anthropic's legal and compliance page; Codex: OpenAI's Terms of Use and Sign in with ChatGPT Terms), say the risk is yours, and name each login variable set in the environment (ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, CLAUDE_CODE_OAUTH_TOKEN, OPENAI_API_KEY, CODEX_API_KEY). Otherwise it prints a notice or nothing, and starts nothing
run [--account name] [--project dir] [--token] [--as-machine [--role orchestrator|worker]] -- claude|codex [args]	runs the agent as an account (default: the agent's default account, set by accounts use) after the same sign-in check; a login that is gone stops the launch with the fix. The environment is the filtered one of the app (no GH_TOKEN, API keys or other credentials from your shell) plus the account's one folder variable on the host; an inherited CLAUDE_CONFIG_DIR or CODEX_HOME is dropped. --as-machine (host accounts only, after machine setup) adds the machine account's variables (see machine enable), with the address read again at the start, so a renamed login is used at once. The session still has no GH_TOKEN: gh acts through the machine's login in its own folder. Claude Code gets --settings with attribution.commit set to the trailers Agent-Host: <host> and Agent-Role: <role> (the role: --role, else worker when an argument is a dispatched worker's first prompt, else orchestrator); Codex has no such setting, so its commits carry no trailers. Nothing is written to git, gh or shell configuration. --token (Claude Code) adds the stored token as CLAUDE_CODE_OAUTH_TOKEN for that launch only, on the host through the environment and in a container through a file of mode 0600 in the account's volume that the launch reads and removes before the agent starts (never on a command line, never written to devcontainer.json, the ledger or a log). With a Claude Code account below its minimum headroom in a window, run shows the finding (account, window, what is left, reset time) and asks Start anyway? (n is no, input that ends before any answer exits 2 without starting; --yes or accounts limits --continue-below on answers yes), then hands the terminal to the agent. Opt-in auto-pick (accounts limits --auto-pick on, off by default) acts only at this point: for a launch without --account whose default account is below its minimum and has a pool, the account of that pool with the most headroom (both windows known and at or above their minimum) runs instead and its name is printed; with none, --when-none stop (default) refuses with the earliest reset time and wait sleeps until it, then starts under that account. Never inside a running session, never outside the pool. After the agent exits, with rotation on for its agent, run checks the account that ran as accounts handover does, and offers the handover only in a terminal (otherwise it prints the command). BenchKeeper has no command that starts a worker: a worker started through run takes the same path
evals cases add --target skill:<name>|rule:<name> --expected <text> [--should-not] [--project dir] [--scope project|user]	records an agent miss as an eval case; the prompt comes from --run <session run>, --transcript, --query or --description. A library skill keeps its name. Without one, the skill is your own: the case is stored under skill:<name>@project-<id> (the skill is in --project, or the project of --run, in a skill folder of any agent) or skill:<name>@user (in a skill folder of your home), and --scope names which one when the library has the same name. A skill that is neither in the library nor found there keeps the plain skill:<name> (a case for a skill installed later; --scope then fails with no such skill). The cases of an own project skill are stored with the project path in your own evals folder
verify [dir] [--project dir]	deterministic checks, no agent session: files match the ledger, expected plugins or skill copies, project community skills in skills-lock.json. Exit 1 on a failure
evals plan|run [--agents a,b] [--model m] [--target t] [--all]	trigger cases (plugins/benchkeeper/evals/ and your own) as real agent sessions in scratch repositories; miss and false-trigger rates per skill, agent, agent version and model. Uses your subscription; plan lists the targets and the number of sessions and starts none, run asks first (--yes answers it) and refuses when CI is set; runs only targets whose skill or cases changed unless --all. --target takes an id as printed, or skill:<name> for an own skill (the library skill of that name when there is one). An own skill's text is part of the target hash, so a changed own skill runs again; its result records scope (project with the project path, or user); one that no longer exists is left out of the plan

Machine-readable output

The commands marked in the first table take --json. One JSON document on stdout, logs and error messages on stderr. Every document is an envelope: schemaVersion (1), command and the keys below, no prose; a failure has error instead of the other keys. Values of environment variables and file contents never appear (knowledge show prints the entries the maintenance skill stored, with token shapes masked, and no file contents; rules show and skills show print the text of the one file they name, and automations export the script with the text of the template's helpers): a change lists its title and path, not its diff, a settings document lists the names of an env block without values, and a file that does not parse is reported as such without the parser message. skills and packs also work without the word list; a command marked no is refused with exit 2. A changing command never prompts under --json: one that would ask a question refuses with exit 2 and changes nothing, and --yes answers the question as it does in the terminal.

Command	--json
doctor, verify, plan, projects plan, history, history show, projects list, projects status, projects scan, projects audit, projects advise, automations list, automations show, automations diff, automations status, automations export, packs list, packs show, packs warnings, skills list, skills show, skills sources, rules list, rules show, rules check, extensions list, settings list, settings get, sessions board, sessions list, sessions cache show, insights, profile export, knowledge show, accounts list, accounts status, evals plan, evals results, inventory, dashboard	yes: what it reads, nothing written (insights works only with --json)
apply, projects apply, projects add, projects remove, projects rename, projects relocate, projects update, settings set, rules disable, rules enable, packs enable, packs disable, packs scope, skills update, skills new, skills edit, skills delete, automations install, automations update, automations remove, undo	yes: what it changed; a question needs --yes. projects update --dry-run changes nothing and needs none
evals run	yes: the plan and, with --yes, the results of the sessions it started
sessions cache set, sessions cache clear	yes: the cache after the change; a question needs --yes
ui, projects confirm-git, profile import, projects create, presets list, presets save, presets remove, restore, evals cases, packs save, packs export, packs import, packs remove, skills reset, skills remove, skills adopt, skills add, skills enable, skills disable, skills cli, projects unshare, sessions record, accounts add, accounts use, accounts remove, accounts track, accounts limits, accounts handover, accounts login, run, knowledge clear, webhook set, webhook remove, webhook status, webhook send, machine setup, machine status, machine enable, machine disable, machine shell, machine remove, containers reconcile	no

Command	Keys
doctor	level (install level or null), adds, findings (level error, warning or info, check, message, fix, and when they apply tag, dir, file, line, scope, rule, repos), summary (error, warning, info)
verify	ok, checks (name, ok, problems)
plan, projects plan	agents and level (plan) or dir (projects plan), changes (id, step, title, kind, destructive, optional, selected, and when they apply path, argv, error); projects plan also skills (added, removed) and warnings
history	runs (id, kind, status, scope, time, changes, and when they apply of, evalCase, target), backups
projects list	projects (path, added)
packs list	packs (id, own, scope global or projects, skills, plugins)
projects scan	root, found (path, registered)
projects audit	dir, ran (checks that ran), findings (id stable for the same subject, signal the group, severity error, warning or info, message, evidence: file and line plus the check's numbers, fix, effort small, medium or large; same shape as insights), skipped (id, group, reason, enable), summary (error, warning, info); errors first, then by id; exit 1 when a finding is an error (none is today). No file content or command line appears
skills list	dir, hasSetup (false: no state file, the lists are empty), skills (name, source, own, state, pinned, newest, packs, single), untracked (name, folders, library), globalUntracked (name, folders)
rules list	dir, agents (id, name, rows: scope, owner, path, shown, bytes, lines, tokens, load, by, switch, as the Rules view; budget: total, largest, over, threshold, notes, limits), limit (tokens of one file), findings as rules check. Sizes and states, never the text of a file
rules check	dir, limit, findings (as doctor, check rules), summary (error, warning). A finding names file and line, never the text; exit 1 on an error finding
extensions list	dir, agents (id, name, rows: kind (mcp, hook, subagent, command, style), name, scope, owner, path, shown, detail, runsCode, and where they apply transport, matcher, state, plugin, removal; problems: files that do not parse). No environment values, headers or file text
settings list	agent, name, layers (layer, path, exists, readOnly, error), precedence (the layer names, highest precedence first), settings (key, value, layer, file, sets, changed, notes, and the key's definition), bundles, notes (each with kind, default, risk and fixChanges), dismissed (the notes dismissed in Agent settings); Codex also trust; with --advanced also advanced, outside and, for Claude Code, env (name, layer, secret, editable). Values of names that look like secrets are masked as (set)
settings get <key>	agent, setting (one entry of settings or advanced)
skills sources	sources (id, title, license, pin, verdict, hooks, limits, evaluation, kind, agents, files)
sessions board	sessions (agent, id, project, branch, agentVersion, start, end, turns, models, costUSD, costEstimated (true when the cost is a Codex estimate), busyMs, tokens, tier, skills, tests, setup, pattern, subagents, run, setupVersion, prs), newest first. Reads the transcripts like the text command but writes no board.html and opens no browser; the session title is left out; with Ponytail's ledger present a row has source (transcript, transcript and Ponytail ledger or Ponytail ledger only, a ledger row with no transcript) and, unless it is transcript, ponytail (costUSD, turns, agentMinutes, reusePct, tests, level), and notes lists the ledger files and rows skipped
insights	samples (sessions Claude Code sessions in the window, minSamples 20), findings (id stable for the same subject, signal unused-skills, friction, hygiene, freshness or skills, severity error, warning or info, message, evidence: the subject's own keys, fix), summary (error, warning, info), and with --narrative narrative (agent, text model text, covers the finding ids it covers, cost as recorded: usd API-equivalent for Claude Code, tokens for Codex, or null; at; cached); errors first, then by id; exit 1 when a finding is an error. No session title, message text or environment value appears
profile export	profile (the profile file: kind, schemaVersion, appVersion, global, packs, projects, each project with origin under --origins); the document is the envelope, so profile import reads only its profile value (extract it from the redirected stdout); a file argument is refused
accounts list	accounts (name, agent, mode host or container, label, default, pool or null, loggedIn as in accounts status, limits (the minimum and warn percentages that apply, per fiveHour and sevenDay), dir for host or volume and mount for container, token: kept and created) and settings (minimum, warn, autoPick, whenNone, continueBelow, rotate: off or handover per agent, Codex always off); no credential is in the registry or the document
accounts status	accounts (name, agent, mode, loggedIn true, false or null when it could not be read, method, tokenAgeDays, usage, findings: level, check, message, fix); usage is null for a container account that was not reached and for a Codex account that is in a container, not signed in or whose app-server read failed (notAvailable says which), else tracked, checkedAt and ageSeconds (null while nothing is kept), minimum and warn (fiveHour, sevenDay percent) and, for fiveHour and sevenDay, state (ok, warn below the notice level, low below the minimum, unknown), used, remaining, resetsAt (ISO time) and reason (why it is unknown), and failures (at, error) of the last 24 hours; exit 1 when a finding is an error (a token past its year), never for low usage or a failed turn
apply, projects apply, settings set, rules disable|enable, packs enable|disable|scope, skills update|new|edit|delete, automations install|update|remove, undo, projects add|remove|rename|relocate	runs (the runs the command made, none when there was nothing to change: id, kind, scope, status, entries: step, title, kind, status done, failed or skipped, and when they apply unit, reason, error); the projects commands also projects (path, status added, removed, renamed, relocated, or for remove --setup the status of that setup removal with run, branch, push and reason where they apply; from for relocated). A command that stopped after a change also has error. No diff, file content or environment value appears; exit 1 when a run did not finish
projects update	as the changing commands; projects has per project path, status (applied, current, committed or stopped) and, where they apply, run, branch, push (the command that pushes the branch), reason. With --dry-run, runs is empty and projects has per project path, setupVersion (null without a state file), ready (the update would change it), reason (the status line) and changes (as in plan)
evals plan, evals run	plan (target, agent, model or null, cases: the case ids, project for a project skill); evals run with --yes also results per target (time, target, agent, agentVersion, model, hash, appVersion, cases, missRate, falseTriggerRate, scope and project for an own skill, details: id, should, exit, and triggered for a skill or review, the session output file, for a rule) and error when a session failed (exit 1); without --yes plan and error (exit 2). No query or expected text appears
history show <run>	run (id, kind, status, scope, time, of for an undo, entries: step, title, kind, status, and when they apply unit, reason, error, path, removed, folder, files with status and path for a recorded session); no diff
projects status [dir]	version (the app), level and adds as in doctor, agents (id, name, tested, installed, version, rules, skills, unavailable, plugin for an agent with plugins: installed, available, for Claude Code accounts), projects (every registered one, or dir: path, setupVersion, current, checkout, origin, repo, branch, clean, changes as in plan or null, error, skills per agent, share, committed, agents, packs, tier; missing for a folder that is gone)
packs show <id>	pack (the pack's definition: id, title, skills or plugins, own for an own pack, and usedBy: the registered projects that name it)
skills list --library	skills (the library entries: name, section, source, license, agents, pickWhen, and projects: the registered projects that have the skill)
sessions list	runs (run, sessions, costUSD or null, tokens, tasks with task and the same counts); with --run, that run only
sessions cache show|set|clear	keepDays (null: kept for good), sessions (the summaries in the cache)
evals results	results (each stored result as evals run adds it, without the per-case details)
inventory	items (step, agent, title, state)
dashboard	global (skills, packs, states), projects (path, name, setup, agents, packs, skills, states, untracked, error), coverage, catalog, as the Dashboard
rules show <file>	path, text (the first 200000 characters), truncated
automations list	templates (id, tier, title, agent, schedule, installed), errors (tier, file, message of a template that does not load)
automations show <id>	template (id, tier, title, agent, schedule, workspace, reuse_session, precheck, missed_run_grace_minutes, mode, parameters, helpers, prompt)
automations export <id>	id, script (the sh script)
automations diff <id>	id, diff (installed against the template, empty when equal), note (why the runner's side could not be read, or null)
automations status	automations (id, name, runnerId, state, changed, enabled, note); exit 1 when one is not current
skills show <name>	name, source, text (the SKILL.md of a skill the app ships or of your own, else null)
packs warnings <id>	pack, warnings (one sentence per pair of the pack's skills whose descriptions overlap)
projects advise [dir]	dir, agent, env (id, reason, or null), packs (id, reason), skills (name, reason)
knowledge show	dir, path, present (the folder exists), files (name, entries: the stored objects, token shapes in text masked), ignored (name, reason)

The exit codes are the same with and without --json.

Exit code	Meaning
0	clean: no finding of level error (warnings and info do not count)
1	a finding of level error (doctor), a failed check (verify), a change that cannot be made (plan, projects plan), a run that did not finish (the changing commands), in the terminal a question answered no; the list commands have no findings and exit 0
2	usage or read failure: an unknown option, command or subcommand, a name that does not exist (a skill, pack, settings key, automation template, run, account, setup.sh backup, or sessions list --run without sessions), a folder with no setup yet for a command that changes a project's setup (packs enable|disable --project, skills update|reset|remove|adopt|add|enable|disable), an option or extra argument the command does not read (<flag> does not belong to <command>, unexpected argument), a bad value, an unreadable file (also the registered projects file, for doctor, plan, projects list, projects status, projects scan, insights and profile export; an unknown --only group or a folder that does not exist for projects audit, doctor, verify and insights --project), --json on another command, a changing command under --json that would ask without --yes, and with --json an uncaught error

Web UI

benchkeeper ui serves on 127.0.0.1 at a free port and opens the browser with a one-time link (--no-open prints it). The link works once and for 10 minutes, and only as printed: at http://127.0.0.1:<port> with the printed port, not at localhost or through a forwarded port with another number. A used, expired or wrong-host link, and a tab left open after the server was started again, show what happened; start the app again (npx benchkeeper@6.26.0) and open the new link. The browser gets a file only you can read, which forwards to the link, so the token stays out of process arguments other local users can see. The link's token becomes an HttpOnly, SameSite=Strict session cookie named per port (bk_session_<port>) and an API key that the page keeps in its own origin's storage. API requests need both: browsers send the cookie to every server on 127.0.0.1, the key stays with this page. The sessions board frame, which cannot send a header, loads with a one-time ticket the page asks for with the key. The server rejects other Host headers, foreign or null origins, cross-site fetches, changes sent with GET, and API requests without the key in the x-benchkeeper header. It stops with Ctrl+C in its terminal or with the Stop button in the header, which the server refuses while an apply or eval runs; scans and setup suggestions still running are cancelled. Recorded agent sessions run from the CLI only. Applies, evals and setup suggestions run on the server: while one runs, the header shows it and opens its progress, also after a reload.

The sidebar groups the views: Overview (Dashboard, History), Projects (Projects), Catalog (Skills, Packs), Analysis (Sessions, Runs, Insights, Evals), Agent configuration (Global setup, Agent settings, Rules, Extensions, Accounts), Machine (Automations, Preferences), and Advanced while Editors is on.

View	Contents
Dashboard	from the top: the status panel (the count of items that need attention with the global setup and each project it counts as links, the projects that cannot be read, Fix everything, and the Set up menu with Set up everything, Set up global only and Set up projects) and the Doctor findings strip (the count of findings, linked to the Doctor section, with the count by level); Health, one bar of item states (current, modified, outdated, missing, removed, foreign, and untracked for a skill folder in a project that BenchKeeper did not install) with each count a link to what it counts; Where, one state bar per scope (the global setup and every project) with its counts and packs; Doctor, the three worst findings, errors first, one line each with its action, and a link to all of them in Insights; folded below them, every finding with the action that fixes one (Plan fix, Review removal, Open settings or Plan project) or the fix by hand, a link to the source a fix names, each run id a finding names linked to History, and Dismiss on a finding that is an opinion; "Dismissed (n)" lists the dismissed findings and declined changes with Show again; Recent runs, the last 6. Folded under "More about the setup": the Setup flow, the install level and what the next adds, the BenchKeeper plugin per agent, the installed skills and packs (global and per project), coverage (the projects that chose each pack, per agent; skills that exist for one agent only are marked; packs no project chose are one line of links), the catalog by pack as a sorted bar list, and the global setup by part and agent (a cell opens that agent's plan). Every figure links to the Global setup, Projects or Catalog view it counts, which opens filtered (Projects by pack and agent, Catalog by pack and agent). One-run setup (buttons on the Dashboard, entries in the command palette): Set up everything, Set up global only or Set up projects, with a level choice for the global setup and a checklist of the projects Update all does not leave alone (a shared setup, one still committed or no repository stay with Update all). One start form and one consent for the agent suggestion sessions (read only, one at a time) of the projects without a setup; one progress bar with a line per item (waiting, running, ready, failed with the reason, cancelled); one review page with a section for the global plan and one per project, a checkbox per section and per change, each diff behind a button, and the project's answers (agents, environment, packs, tier) editable in place, which builds its plan again; then one apply job with progress per section and a link to each run in History for undo. A failed suggestion or plan shows its reason and leaves the rest, a failed section leaves the others. Cancel during preparation stops the running suggestion session and keeps what is ready for review; during the apply it stops after the change that runs. Fix everything walks through four steps (the global plan, the projects behind or with pending changes, none checked at first, the doctor findings left to do by hand, a summary) and then applies the global plan and runs Update all for each checked project; "Stop after this project" skips the rest. A global setup that fails or is cancelled leaves the projects unless you start them anyway; the doctor runs again at the end. It stays open over other views until the run is over
Projects	registered projects as cards (the default) or a table, the choice kept in the browser. A card: name and folder (cut at its start), a health dot (in sync, outdated, modified, folder not found, not set up) with the count of doctor findings, the setup in one line (agents, default tier, workflow owner) with its packs, sessions per day of the last 30 days as a bar graph with a text alternative (all agents and the repository's worktrees; the total with its change against the previous 30 days, the cost with the count of sessions without one, the last session and its tier), and Details, Open board (the Sessions view), Set up or Apply (the project's plan), Update and Remove setup. The name or Details opens the project's page: name, health dot and folder, Change setup, a More menu with Open folder, Edit name and notes, Relocate and the two removals, and the tabs Overview (the card's facts, its items by state, the project doctor), Installs (one switch each for packs (in use or suggested as rows, the others as a two-column grid with scope and state under Details), the workflow owner, skills and agent files, with a scope hint: only this project or also global, a scope an agent fixes with its reason, the state on disk, and summary, plain words, impact, default and current; a read-only block of what the global setup makes active here (global packs the project does not select, the BenchKeeper plugin, global skills) with a link to Global setup; the plan below, which alone changes the project, with its diff and undo; Set up (when the project has no setup yet); Skill states: each skill of the project with its state, commit installed and, after Check for updates, the source's newest, a switch to deactivate and activate, Update, Reset, Remove, filter by state, each action opening its plan with the diff, then apply and undo, and untracked folders with Adopt for library skills, left alone otherwise), Sessions (the board with only its sessions), Rules and settings (its instruction files and its project and local settings files) and History (its runs). Above them the totals of the projects shown (projects, in sync, need attention, sessions in the last 7 days) and one health bar. The table: one row each: name and tree state, then path (cut at its start) and branch; one setup status (current, behind or not set up, and local or shared); skills each session of a chosen agent loads (the project's own with its global and plugin skills); pending changes; on the project's page its plan, pack switches (the packs in use or suggested by its files first, then the others, each with when to pick it; with Copy packs from project; the preview lists the skills a pack takes along or removes), skill switches and doctor findings; add (another worktree of a registered repository is offered, not added; a folder inside a project is flagged), scan a parent folder, New from preset (starts a new folder from a saved preset, shows the plan, registers the folder and then recommends skills to turn off and packs to add, none chosen), search, filter chips (needs attention, no setup, folder not found, each pack and agent in use; every chip on must hold) and a sort (name, health, last change), display name and notes, Relocate for a folder that moved (a missing folder shows Folder not found with Relocate and Remove), Remove from list and Remove BenchKeeper setup (every file listed first; a local setup is a run with undo in History, a shared one a commit on a local branch), the same four plus Update and Set up for several selected projects at once (the bar shows only with a selection) with one confirmation naming the count, Update all with progress per project and, per committed branch, the command that pushes it. A project without a state file is set up step by step (agents with where the setup lives: on this machine, the default, or in the repository as a local branch with one commit, which you push; environment with the dev container as default, packs and skills with suggestions from its files, tier, review), one project after the other; Prefill all with an agent, on the first project of a batch, runs one suggestion session per project after one consent, one at a time with a state per project (waiting, running, ready, failed, cancelled); each project's steps then start from its suggestion, still reviewed and applied one by one (a reload loses suggestions not yet used). Change setup reopens these steps from the state file; Open folder shows the project in the file manager. Suggest with an agent asks first, then runs one read-only session (Claude Code -p --restricted --strict-mcp-config with Read, Grep and Glob; Codex exec with a permission profile that reads only the project, the system's minimal paths and Codex itself, writes nothing, and runs without web search and MCP servers) in the project on your subscription; only library packs, skills and environments from its answer are taken. The steps only the user can start (the setup skill session, a pack's own setup) show in the review step, the result step and the project doctor as a Start in a terminal block: agent, project folder and the full benchkeeper sessions record --project <dir> -- <agent> '<prompt>' line with Copy. The app does not run the command: paste it in your own terminal; the recorded run shows in History with its exit code and changed files, and Check again reads the project doctor again, whose warning is gone once the files the step writes exist
Skills	the Catalog's library skills and your own (marked "yours") with filters and the library packs that name each skill as badges (skills list --library), skill detail with eval results (skills show), pack builder (packs save, warns about overlapping descriptions). New skill, and Edit and Remove on one of yours (on its row and its detail), open its SKILL.md text in a dialog and show the plan with its diff (skills new, skills edit, skills delete). A skill's state in a project is on the project's page
Packs	the Catalog's library packs and own packs with the projects that use them (packs list, packs show), export, import (asks before it replaces an own pack, the old file goes to backup/packs/) and removal (off while a project uses the pack). A pack is defined once and switched on per project, on the project's Installs tab (packs enable --project)
Global setup	only global things: detected agents; what is installed globally by part (plugins and packs, skills, rules, settings, machine tools, other), each part with its count, the items that need attention and its state per item behind Show items, and a link to the view that edits it (Extensions, Rules, Agent settings); a box on each item that needs attention (a removal starts unticked), and the button "Plan N items" starts the plan with the ticked ones, a tick after that drops the open plan; a link to Accounts; the line "Per-project setup lives in each project" with a link to Projects (per-project controls are in a project's Installs tab); and the global plan for the chosen agents. A skills CLI install in a plan (here, in a project's plan and in setup review) runs first, on a click, in a scratch folder with its own HOME and CODEX_HOME, CLAUDE_CONFIG_DIR and XDG_* unset, so the plan shows the files it would write (mapped back to the folders those variables point to); links are copied as what they point to, npm uses a scratch cache and your .npmrc without credentials, git an empty config. It runs inside an OS sandbox (srt; bubblewrap on Linux and WSL2 and sandbox-exec on macOS confine writes only where srt cannot run) that stops a write outside the scratch folder and, with srt, limits the network to the registry, GitHub and your registry host. Where none starts (Windows has none) the preview is off and says why; "Rehearse anyway without a sandbox" runs it with the scratch HOME only. doctor names what to install. Three run at a time; a result is kept 10 minutes or until the app changes files, and an apply stops a rehearsal still running. Apply shows each change as it runs, with the command output, then the run report in place of the plan. Cancel skips the changes not started yet
Sessions	the sessions board (see sessions board), scanned when the view opens, with progress and Rescan; each scan also rewrites board.html, which the status band's board button opens. Ponytail's ledger, when present, adds its figures and ledger-only rows (see docs/setup.md). The per-day chart shows all series of a day on hover, tap or with the arrow keys; a click on a project in the rail filters the whole page. In the app the filters stand in a column left of the figures, the project choice first as one chip that opens the list (registered repositories first, one entry per repository with its worktrees folded in, with a search); at phone width one Filters button opens the column. Four figures lead, the others fold under More figures. When Ponytail's own board page (board.html next to its ledgers) exists, Open Ponytail's board opens that file in the browser; no other path is opened
Runs	read when the view opens and on Refresh, no timer: the orchestrated runs of the sessions scan in the registered projects and their worktrees, one row per run and per task with worktree, branch, sessions, cost and tokens; each project's last setup change from the ledger; and, when Orca is installed, its runs with task counts per state (one run-list call and one task-list call per run, at most 20 runs). It shows ids, states and counts only, never a prompt, spec, title or worker output
Insights	the findings of insights: a line with their count by level and a "Do first" link to the most severe, what was checked, then the findings grouped by level (Errors, Warnings, Info; Info folded) and inside a level by area (Setup, Agent settings, Updates, Skills, Sessions), most severe first. An error is a card with its evidence and fix, where the finding has them, and the Dismiss and Snooze 7 days buttons; a warning or info finding is one line with its fix button, where it has one, and a More menu that holds Dismiss and Snooze 7 days, and its title opens the evidence and the fix hint. A doctor finding the setup can fix has the Dashboard's action (Plan fix, Review removal, Open settings or Plan project), which opens its plan with the diff and undo or the view that fixes it; a skill proposal's Plan opens the plan of that project for those skills, with its diff, apply and undo; the others show their fix text. Dismiss hides a finding until its evidence changes (the failed sessions of a project, the unused skills, a doctor finding's details; not the counts), Snooze 7 days until that date or a change, and "Dismissed and snoozed (n)" lists them with Show again. The choice is kept per finding in insights-triage.json in the benchkeeper config folder (readable only by you): finding ids with a digest of the evidence, and dates; no folder name, session id or skill name. The CLI does not read it. Above the findings, Adviser narrative: a summary of the findings shown with a suggested order, from one session of an installed agent (Claude Code or Codex; no tools, an empty folder). It starts only on a click, after a consent text shown once (kept in insights-narrative.json in the benchkeeper config folder); the exact input, with its size in characters, is on the page beside the button, and a run is refused when the findings changed since it was shown. The text is marked as model text, shows its cost as the agent recorded it (Claude Code API-equivalent USD, Codex tokens), is read-only (it cannot apply, dismiss or snooze), and is kept in insights-narrative.json in the benchkeeper cache folder (readable only by you) for that findings set; a changed set (a dismissal included) drops it. It does not run in CI
History	runs by day; the run report: counts by status and changed lines, each change with its diff (line numbers, unified or split, created and deleted files unified), command and output, or session files, undo of a step or the whole run
Evals	per agent and model (the newest by default): the latest miss and false-trigger rate of each target, worst first, and each target's rates over time; Run evals plans the targets and the number of agent sessions, then starts them with progress per session; Cancel stops the running session, finished targets keep their results
Agent settings	the agents' own settings files, apart from Preferences below (the app's own). Per agent (Claude Code, Codex) and layer (user file, project or local file of a registered project): the basic list grouped by topic, one row per setting with its key, one sentence, the value in the layer and the layer in effect (managed settings shown read-only), the default and BenchKeeper's suggestion (settings that differ from the suggestion first, matching ones folded unless a search or "Changed from default" narrows the list; Details opens what a setting does and why it is suggested; a value the global plan offers unselected is labelled "BenchKeeper recommendation"), suggested values for unset keys with their reason and source, notes titled by kind (Conflicting settings: one setting disables another; Hardening: advice against a default; Ignored setting; Note), each with what the default is, what it risks, what the fix changes and the source, and "Dismiss, I accept this" (a "Dismissed (n)" line shows them again), Hardening notes folded under "Hardening advice (n)" with the keys they name, a line counting the settings with a suggestion that differ from it in the chosen file, a "Differs from suggested" marker on such a key with "Use suggested", which plans the suggested value, and "Keep mine", which lists the setting under "Decided (n)" and silences it here and in Doctor until Show again, search with the switches "Changed from default" and "Differs from suggested" (both filter the basic list and Advanced), the "less noise" switches as one plan, and, for Codex, Trusted projects (each folder of the user file with its trust and whether it still exists; Remove marks one that no longer exists for removal); Advanced lists every key of the pinned schemas, the environment entries (names of secrets never with a value) and keys outside the app's list, which edits keep. Review shows the plan with each diff; Apply writes through the same runner, History undoes it
Rules	a summary strip (per agent, the tokens that load at the start of every session against the limit, the largest source, and the checks), then one tab per agent (Claude Code, Codex) with its instruction files, globally or for a registered project (its own folder only: Claude Code also loads the CLAUDE.md files of folders above it, which the view does not list), in load order: scope, owner (a file BenchKeeper did not write is yours: nothing here deletes it, and Doctor lists its rules check findings except a missing @import; it reports drift from the setup only for the files the full level writes its rules into, ~/.claude/rules/00-global.md and ~/.codex/AGENTS.md, whoever wrote them), size, whether it loads (on, only for matching files, off, or replaced by a Codex override) and, where available, a switch for a file BenchKeeper did not write, as a plan with a diff and undo. A file's text opens read-only in a drawer on this machine only; Edit there (files of yours and the project's) counts tokens and lines and re-runs the checks while you type, and Save is a plan with the diff, a backup in the run and undo in History. The context budget adds rule files, @ imports, skill descriptions and auto memory (first 200 lines or 25 KB of MEMORY.md), at four bytes per token, names the largest source and warns above 10,000 tokens (BenchKeeper's own limit); Codex notes its 32 KiB cap. Claude Code's auto memory folders with notes show read-only, per project (all of them in the global view): the topic notes, the total size, the last change and whether MEMORY.md is over the load limit; never the text. Codex files that ship with a repository, or a global Codex file for one project, have no switch: Codex has no setting for it
Extensions	the MCP servers, hooks, subagents, commands and output styles per agent, globally or for a registered project, read from files only: owner, the file, a "runs code" mark for hooks with a command, stdio servers and commands that run a shell line, and a link to the removal offer of the global plan (full level) for a global server or hook. No switch: it only shows. Environment values and headers never appear
Accounts	the accounts per agent (Claude Code, Codex) on the host or in a dev container, each with its sign-in state (Check), a default switch, Sign in and Remove; Add account as a plan with a diff and undo. Sign in is the guided flow of accounts login: the steps (account, private window, code if the browser shows one, check), the link and the agent's own lines, with a field for the code. The page offers no token: it shows the command. Check also shows the usage left in the 5-hour and 7-day window (or why it is unknown, or the command that starts tracking it) and the usage notices. A Usage limits section shows the minimum headroom per account, the auto-pick setting and the provider's terms caveat; the values are changed with accounts limits. It states the container risk (a hijacked session can read the credentials in the volume) and that nothing switches accounts mid-session
Preferences	three tabs. Views: views and their cost (the default of the terminal tier strip, the sessions board, Sessions and Runs and what each runs while on; the tier strip switch writes tierBand in the Claude Code user settings as a plan with undo, off writes false; one status line select per host account that accounts track set up writes the plugin option statusStyle in that account's settings.json, the file its status line reads, as a plan with undo), keyboard shortcuts (the / key can be switched off). Sessions history: how many days the sessions cache keeps summaries of deleted transcripts. Advanced: profile export and import, the switch Editors (editors in preferences.json, off by default), config files and, last, Clear the sessions cache
Orchestrator policy	under Advanced, only while Editors is on in Preferences (off: no view and no route). Per registered project, docs/agents/orchestration.json as JSON text, checked against plugins/benchkeeper/schemas/orchestration.schema.json while you type (each error with its rule and path). The text is written as typed, so unknown and // keys stay. Review shows the plan with its diff; Apply writes through the runner, History undoes it

The view, a project's page and its tab are part of the URL (#/projects/<dir, encoded>/sessions), so a reload and Back keep them; an older #/projects?path=<dir> link opens the project's Installs tab, and a folder that is not registered shows a link back to Projects. "Choose folder" opens the system dialog: osascript on macOS, zenity, then kdialog on Linux. With none of them, on Windows, on Linux without a display, or after a cancel, type the path. The server blocks other requests while the dialog is open.

Keys	Action
Ctrl+K (macOS: Cmd+K), /	command palette: views, projects, skills, the last 20 runs, plan global setup, run doctor, toggle theme
Ctrl+1 to Ctrl+9 (macOS: Cmd)	the first nine views in sidebar order: Dashboard, History, Projects, Skills, Packs, Sessions, Runs, Insights, Evals
Ctrl+J (macOS: Cmd+J)	light or dark theme

Where the browser handles one of these keys itself, the sidebar, the palette and the header buttons do the same.

The UI is built from dev dependencies (Mantine core, charts and spotlight, Recharts, Tabler icons, the Inter font, React) into dist/. The published package has no extra runtime dependencies.

Routes and commands

Every route of the UI server (src/server.mjs) with the command that does the same, or one reason it has none. A test fails when a route is missing here or a row names a route that no longer exists.

Reason	Meaning
UI internal	only the page needs it (job control, tokens, dialogs, form options, previews); no command planned

Route	Command or reason	Note
GET /	ui	the page
GET /board	sessions board
GET /api/overview	projects status	agents, level, plugin state (for Claude Code also per registered account: installed version and whether it is behind) and setup version per project
GET /api/global-plan	plan
POST /api/apply	apply
POST /api/apply-all	apply, projects apply	the UI runs the global and the project plans as one job, each its own run
POST /api/prerun	UI internal	rehearses a plan's skills CLI commands in a scratch folder inside an OS sandbox; body unconfined: true runs them without one; plan prints the command lines
POST /api/job-cancel	UI internal
GET /api/jobs	UI internal
GET /api/job	UI internal
GET /api/eval-plan	evals plan	the plan, then the question
POST /api/eval	evals run
GET /api/projects	projects status	setup state per project; the page also gets the skills each agent sees and the diffs of the pending changes
GET /api/presets	presets list
GET /api/create-plan	projects create	the plan of the new folder; the page registers it with POST /api/projects after the apply
GET /api/recommendations	projects create	the check after creation: the recommendations with their reasons and their plan, no change selected
GET /api/project-check	projects add	add refuses what the check reports
POST /api/projects	projects add, projects remove, projects rename, projects relocate
GET /api/remove-setup-plan	projects remove --setup	the files are listed before the question
POST /api/remove-setup	projects remove --setup
GET /api/project-plan	projects plan
GET /api/skills	skills list	per project; dir keeps one
GET /api/project-setup	UI internal	fills the setup form; packs list, skills sources and help show its lists
GET /api/one-run	UI internal	start state of the one-run dialog
POST /api/setup-advice	projects advise
POST /api/board-ticket	UI internal
POST /api/ponytail-board	UI internal	opens Ponytail's own board file with the system opener; 400 when it is missing
POST /api/shutdown	UI internal
POST /api/sessions-scan	sessions list, sessions board	both scan; sessions board --json prints the rows
GET /api/sessions-cache	sessions cache show
GET /api/tier-band	UI internal	the tier strip setting's key and state; the Preferences switch plans it through POST /api/settings-plan
GET /api/status-style	UI internal	the status line style setting's key and, per host account that accounts track set up, its name, the settings.json its status line reads and the style set there (merged when unset)
POST /api/status-style	UI internal	{ account, value }: a plan that sets the style in that account's settings.json, applied through POST /api/apply
GET /api/runs	sessions list	adds each task's worktree and branch, each project's last setup change and Orca's runs
GET /api/project-cards	UI internal	per registered project: sessions of the last 30 days, doctor finding count, items by state, workflow owner packs
POST /api/sessions-cache	sessions cache clear, sessions cache set --keep-days	the command cannot see a scan the web UI runs: clear while none runs
GET /api/insights	insights --json	adds key and triage (null, dismissed or snoozed until a date) to each finding
POST /api/insights-triage	UI internal	body key and action: dismiss, snooze (7 days) or reopen
GET /api/insights-narrative	UI internal	consent state, installed advisers, the shown findings' count, the exact prompt with size and id, and the cached narrative for this set
POST /api/insights-narrative-consent	UI internal	records the one-time consent
POST /api/insights-narrative	UI internal	body agent and id of the reviewed input; a job (refused without consent, in CI, or when the findings changed)
POST /api/update-all	projects update
GET /api/history	history
GET /api/run	history show	one run with the diff of each file; under --json the files without their diffs
POST /api/undo	undo
GET /api/catalog	skills list --library, packs show	library skills and packs with their users
GET /api/skill	skills show
POST /api/user-skill-plan	skills new, skills edit, skills delete	body op (new, edit, remove), name and text; a plan applied through /api/apply
GET /api/doctor	doctor	?dismissed=1: the findings dismissed or declined in preferences.json
POST /api/pick-folder	UI internal
POST /api/reveal	UI internal
GET /api/evals	evals results
GET /api/pack-warnings	packs warnings	the page sends a skill list, the command takes a pack
GET /api/pack-export	packs export
POST /api/packs	packs save, packs import, packs remove
GET /api/profile-export	profile export
POST /api/profile-import	profile import
POST /api/scan	projects scan
GET /api/settings	settings list, settings get
POST /api/settings-plan	settings set
GET /api/accounts	accounts list	without loggedIn (each card reads it through the status route), also the usage settings, the credential store in use and a sign-in job still running
GET /api/accounts-status	accounts status	with the usage and, for Claude Code, the address and plan when its status gives them; accounts limits is a command only
POST /api/accounts-plan	accounts add, accounts use, accounts track, accounts remove	an account that keeps a token is removed by the command, which deletes the token; a dev container account is tracked by the command
GET /api/automations	automations list, automations status	per template its fields and parameters, per installed one its state, values, project folder, last run and skipped runs as the runner reports them
POST /api/automations-plan	automations install, automations update, automations remove	install only into a registered project
POST /api/accounts-login	accounts login	the job opens the sign-in link in a private browser window or shows it; --token is the command only
POST /api/accounts-add	accounts add, accounts login	add by sign-in: signs in first in a new default folder, then registers the account under a name from its identity
POST /api/accounts-code	UI internal	the code the browser shows, handed to the agent and kept nowhere
GET /api/rules	rules list
GET /api/extensions	extensions list
GET /api/rule-text	rules show
POST /api/rules-plan	rules disable, rules enable
POST /api/rule-check	rules check	the size and findings of a rule file with the text being edited, nothing written
POST /api/rule-edit-plan	UI internal	a plan that writes the edited text of a rule file of yours
POST /api/preferences	UI internal	the app's own preferences (editors)
POST /api/settings-dismiss	UI internal	dismiss an Agent settings note, or keep a setting as decided (setting:<agent>:<key>), a declined global change (change:<step>:<hash>) or a Doctor finding, or show it again (dismissed in preferences.json); Doctor leaves out a dismissed note and a note whose settings are all decided
GET /api/params	UI internal	the policy file with its errors; 404 while editors is off
POST /api/params-check	UI internal	404 while editors is off
POST /api/params-plan	UI internal	404 while editors is off
GET /api/inventory	inventory
GET /api/dashboard	dashboard

Configuration

Name	Default	Effect
BENCHKEEPER_HOME	your home directory	the home the app reads and writes; tests point it at a temporary directory
BENCHKEEPER_DEBUG=1	unset	a failing command prints its stack, masked, instead of only its message (the ui server always writes the stack of a failure to its terminal)
BENCHKEEPER_NO_UPDATE_CHECK=1	unset	no registry request for a newer version (it runs only in a terminal, not when CI is set, and never installs)
BENCHKEEPER_SKILLS_CLI=x.y.z	unset	runs exactly that skills CLI version, which must be at least 7 days old (the install keeps the 7-day cutoff, so a newer one fails instead of running); unset, the newest release at least 7 days old, never below the app's minimum

Files below ~/.config/benchkeeper/: ledger/ (one JSON file per run), backup/<run>/ (replaced and removed files), projects.json, accounts.json, preferences.json (the app's own preferences: editors turns on the editor view, dismissed lists the Agent settings notes you dismissed, the settings you decided to keep, the global changes you left unselected in an apply and the Doctor findings you dismissed; while the file does not parse, BenchKeeper reads it as the defaults and writes nothing to it: a dismiss or the Editors switch fails naming the file, and an apply runs without remembering what you left unselected, with a warning), packs/, skills/ (your own skills, one folder each), evals/ (own cases, results.jsonl, outputs of rule cases), automations/ (own automation templates), automations.json (the installed automations), automations-installed/ (copies of the precheck helpers they call), board.html, pack-scopes.json (the install scope chosen per pack), disabled-rules/ (global rule files turned off in the Rules view), kept-rules.json (global rule files whose edits you keep), insights-triage.json (dismissed and snoozed Insights findings), insights-narrative.json (the consent to Insights narrative sessions), orca/ (the optional Orca worker spec and orca.yaml and .worktreeinclude samples, written when chosen and Orca is installed), and machine/ (the machine account: machine.json with both accounts' ids and logins, never a token; gh/, the gh config folder it is signed in to; signing_key and signing_key.pub, its ssh signing key, mode 0600). Caches below ~/.cache/benchkeeper/: worktrees/, sessions.json, insights-narrative.json (the last Insights narrative), webhook.log (webhook deliveries).

Test

In the dev container:

• npm test: unit tests against temporary home directories with stubbed agent commands
• npm run smoke: start, plan, apply and ledger write
• npm run build && npm run test:ui: the web UI against a fixture home, axe in light and dark mode

Release

The app (this package, tag app-v<x.y.z>) and the content (plugins/benchkeeper/, tag setup-v<x.y>) release as two components with their own versions (releasing); test/package.test.mjs checks that the content's minApp is not above the app version. npm pack copies rules, own skills, the library catalog and eval sets into content/ and builds the UI, so plan and apply work offline for them. After the owner merges a release pull request that raises the app version, the Release workflow starts the "npm publish" workflow (npm.yml) with the new version and checks the registry (releasing); a release of the content alone publishes nothing to npm. A manual run of npm.yml (Actions, Run workflow, the version as input) is the fallback. It publishes through npm trusted publishing (no token) with provenance. The trusted publisher in the package's npm settings names this repository and npm.yml.