the-benchmark
Advanced tools
Minimal zero-dependency benchmark for local LLM servers (LM Studio, Ollama, llama.cpp, vLLM): prefill, generation, a live concurrency sweep, and a real agentic tool-calling loop.
A local LLM benchmark for measuring prompt processing, generation, concurrent requests and tool calls, with an optional Studio for independently graded coding tasks.
Load a model in Ollama or enable LM Studio’s local server, then run:
npx -y the-benchmark
# Or with Bun:
bunx the-benchmark
The CLI requires Node.js 18 or later, or Bun 1.4 or later. It detects a running local backend; pass --target ollama or --target lmstudio for unattended runs when both are available. Other OpenAI-compatible endpoints can be selected explicitly:
npx -y the-benchmark --url http://localhost:8080
npx -y the-benchmark --phases concurrent --concurrency 1,2,4,8
npx -y the-benchmark --phases agentic
For the local UI:
npx -y the-benchmark studio
Open localhost:4310. Studio requires Node.js 22.22 or later and npm; its first launch installs the optional UI. Independently verified tasks also require Docker and internet access for initial runtime setup. Native Linux and Windows verified-task execution are not yet validated; see the Studio guide.
Both modes work directly from the package, without cloning or a global install:
| Mode | npm | Bun |
|---|---|---|
| CLI only | npx -y the-benchmark | bunx the-benchmark |
| Local UI | npx -y the-benchmark studio | bunx the-benchmark studio |
| Help | npx -y the-benchmark --help | bunx the-benchmark --help |
In Studio, Compare models charts saved verified-task runs by task. Select one run per model; selecting a newer attempt replaces that model's previous selection. The charts show passes out of all planned attempts and median time for passing attempts, with configuration differences and errors visible alongside the results.
The three bundled verified tasks are an integration suite, not a broad coding leaderboard. Compare runs with the same tasks, budgets, server configuration and hardware.
For development, clone the repository and install with bun install --frozen-lockfile. These commands run from the checkout:
| Command | Description |
|---|---|
bun run bench | Run the default CLI benchmark |
bun run concurrent | Run a concurrency sweep |
bun run agentic | Run the tool-calling smoke test |
bun run studio:setup | Install and build the optional Studio UI |
bun run studio | Start Studio |
bun run studio:test | Test the Studio runner and API |
bun run test | Run CLI regression tests |
bun run check:ci | Run the complete repository validation contract |
See the CLI reference, Studio guide, decision-model guide and recorded results. The column naming proposal is a design discussion, not current CLI behavior.
For maintainer validation, see QUALITY.md. Bun’s release-age policy can select an older eligible CLI; pin a version to check it explicitly, wait for it to age, or use npm. Reference guides are linked from GitHub so they remain accessible from the npm package.
Ryzen 7 5800X3D / RX 7900 XT with LFM2.5 24B on Ollama: 152.5 generation tokens/s. The separate three-task verified suite passed 1/3 at 32k output tokens per turn. See results and budget comparisons and the archived samples, artifacts and charts.
MIT — see LICENSE.
FAQs
Minimal zero-dependency benchmark for local LLM servers (LM Studio, Ollama, llama.cpp, vLLM): prefill, generation, a live concurrency sweep, and a real agentic tool-calling loop.
We found that the-benchmark demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Security News
/Application Security
A practical guide to securing GitHub Actions against supply chain attacks, covering malicious pull requests, credential theft, dependency risks, and release pipelines.
Security News
Composer 2.11 adds dependency cooldowns, joining package managers that delay new releases to reduce supply chain risks while allowing urgent security updates.
Threat Research
/Security News
Cursor removed a malicious Claude Code 2.1.313 extension from its marketplace. Socket examines registry records and the initial claims about its distribution.