inferbench-cli
Advanced tools
inferbench-cli - npm Package Compare versions
+6
-3
| { | ||
| "name": "inferbench-cli", | ||
| "version": "0.1.3", | ||
| "description": "Benchmarks local-LLM-inference engines (omlx, llama.cpp) on your own hardware, live, and recommends the fastest engine/config combination for your model.", | ||
| "version": "0.1.4", | ||
| "description": "Benchmarks local-LLM-inference engines (omlx, llama.cpp) on your own hardware, live, and recommends the fastest one for your model.", | ||
| "keywords": [ | ||
@@ -14,3 +14,6 @@ "llm", | ||
| "apple-silicon", | ||
| "cli" | ||
| "cli", | ||
| "gguf", | ||
| "tokens-per-second", | ||
| "hardware-benchmark" | ||
| ], | ||
@@ -17,0 +20,0 @@ "homepage": "https://github.com/RudrenduPaul/InferBench#readme", |
+107
-47
| # InferBench | ||
| Every "best local LLM engine" article benchmarks someone else's machine. InferBench benchmarks yours. | ||
| [](https://github.com/RudrenduPaul/InferBench/actions/workflows/ci.yml) | ||
| [](https://www.npmjs.com/package/inferbench-cli) | ||
| [](https://pypi.org/project/inferbench-cli/) | ||
| [](./LICENSE) | ||
| Every "best local LLM engine" article benchmarks someone else's machine. InferBench benchmarks yours. | ||
| ```bash | ||
| npx inferbench-cli run --engines llama.cpp --model "bartowski/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M" | ||
| ``` | ||
| --- | ||
| Local-inference engines all publish their own benchmarks, on their own hardware, in their own README. None of them tell you which one is actually fastest on the machine sitting in front of you. InferBench runs a fixed, varied prompt set against whichever supported engines are installed on your own hardware and reports real, measured tokens/second -- not a number copied from someone else's blog post. | ||
| ## What it does | ||
|  | ||
| ```bash | ||
| $ inferbench run --engines llama.cpp --model "bartowski/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M" | ||
| Hardware: Apple M4 (darwin/arm64), 16GB | ||
| llama.cpp: starting server... | ||
| llama.cpp: warming up... | ||
| llama.cpp: [1/8] benchmarking... | ||
| ... | ||
| llama.cpp: [8/8] benchmarking... | ||
| Results: | ||
| llama.cpp: avg 75.54 tok/s (range 69.54-79.78, n=8) | ||
| Recommendation: llama.cpp -- highest measured throughput on this run (75.54 tok/s avg) -- specific to this hardware and model, not a universal ranking | ||
| npx inferbench-cli run --engines llama.cpp --model "bartowski/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M" | ||
| ``` | ||
| Every number above is real, produced by a live run against an actual `llama-server` process on real hardware -- not an illustrative placeholder. | ||
| ## Table of contents | ||
| ## Why this exists | ||
| - [Install](#install) | ||
| - [Features](#features) | ||
| - [Quickstart](#quickstart) | ||
| - [CLI command reference](#cli-command-reference) | ||
| - [Library API reference](#library-api-reference) | ||
| - [How the measurement works](#how-the-measurement-works) | ||
| - [Comparison](#comparison) | ||
| - [Why this exists](#why-this-exists) | ||
| - [Documentation](#documentation) | ||
| - [FAQ](#faq) | ||
| - [Contributing](#contributing) | ||
| - [Security](#security) | ||
| - [License](#license) | ||
| Local inference on consumer hardware is now the default path for a growing share of developers, and every engine's own comparison against its competitors has an obvious incentive problem: no vendor is a disinterested judge of its own numbers. InferBench has no engine of its own to sell, which is the entire point. | ||
| The harder question this tool actually answers isn't "which engine is fastest in general" -- there is no such answer, because it depends on your exact hardware, your exact model, and your exact workload. It's "which engine is fastest **right now, on this machine, for this model**" -- a question only a tool that runs on your own hardware can answer honestly. | ||
| ## Install | ||
@@ -59,3 +51,3 @@ | ||
| **Current status**: both packages are published and installable today. | ||
| Both packages are published and installable today. | ||
| `npm install -g inferbench-cli` and `pip install inferbench-cli` both | ||
@@ -77,2 +69,12 @@ work -- see | ||
| ## Features | ||
| - **Cross-engine, same measurement code.** InferBench starts each engine's own OpenAI-compatible HTTP server (`llama-server`, `omlx serve`) and sends every engine the identical prompt set through the identical timing code, instead of comparing numbers each engine's own benchmark tool produced differently. | ||
| - **Full-response-body timing, not headers.** An earlier version of this code measured elapsed time right after the HTTP response object resolved, which only captures headers arriving, and once reported a physically impossible 64,646 tok/s before the bug was caught. Both distributions now time the complete response body, with a regression test guarding the fix in each language's harness. | ||
| - **8-prompt fixed sweep with warm-up.** One throwaway completion absorbs first-request latency, then 8 varied prompts are timed individually and reported as avg/min/max tok/s (`n=8` in the results table). | ||
| - **Two independently maintained distributions, matching output.** npm's `inferbench-cli` (TypeScript) and PyPI's `inferbench-cli` (a genuine Python port, not a wrapper around the Node binary) expose the same CLI flags and the same JSON report field names. | ||
| - **Machine-readable reports.** `--json` / `--out <file>` writes a full `BenchmarkReport` as camelCase JSON on both distributions, so CI or an agent can parse it without special-casing which language produced it. | ||
| - **Local cloud-cost context (Python library).** `compare_to_cloud()` looks up a static, dated cloud API price alongside your measured local throughput -- it discloses plainly that it's a snapshot, not a live quote, and returns `None` for a model it doesn't recognize rather than guessing a number. | ||
| - **Path-safe `--out`.** A relative `--out` value that resolves outside the current working directory is rejected, so an agent-supplied output path can't escape the intended directory. | ||
| ## Quickstart | ||
@@ -95,2 +97,20 @@ | ||
| Real output from a live run against an actual `llama-server` process: | ||
| ``` | ||
| $ inferbench run --engines llama.cpp --model "bartowski/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M" | ||
| Hardware: Apple M4 (darwin/arm64), 16GB | ||
| llama.cpp: starting server... | ||
| llama.cpp: warming up... | ||
| llama.cpp: [1/8] benchmarking... | ||
| ... | ||
| llama.cpp: [8/8] benchmarking... | ||
| Results: | ||
| llama.cpp: avg 75.54 tok/s (range 69.54-79.78, n=8) | ||
| Recommendation: llama.cpp -- highest measured throughput on this run (75.54 tok/s avg) -- specific to this hardware and model, not a universal ranking | ||
| ``` | ||
| **Known v0.1 limitation, stated plainly:** `--model` means something different per engine (a downloadable HF spec for llama.cpp, a pre-downloaded local directory name for omlx), because the two engines have genuinely different model-acquisition capabilities -- omlx's `serve` command has no flag to pull an arbitrary model from Hugging Face directly. Running both engines against the *same* model in one command therefore needs the model already available in both engines' own expected forms. | ||
@@ -113,4 +133,42 @@ | ||
| Exit code `0` on a successful run with at least one engine tested; `1` on a usage error or when no supported engine is installed. | ||
| Exit code `0` on a successful run with at least one engine tested; `1` on a usage error or when no supported engine is installed. The Python CLI has one small, documented divergence: a missing required `--model` flag exits `2` (the standard `argparse` convention for a parse-time error) instead of `1`. | ||
| ## Library API reference | ||
| The Python package (`pip install inferbench-cli`) exposes a documented library surface, meant for use in scripts or notebooks instead of the CLI. The npm package's `package.json` `main` field points at the CLI script itself (`dist/cli.js`, which runs the argument parser as a side effect on import) and does not declare a separate library entry point, so today only the Python distribution is a supported library import. | ||
| ```python | ||
| from inferbench import ( | ||
| benchmark_engine, detect_hardware, all_engines, resolve_engines, | ||
| recommend, compare_to_cloud, report_to_dict, write_json_report, | ||
| ) | ||
| ``` | ||
| | Symbol | Signature | What it returns | | ||
| |---|---|---| | ||
| | `detect_hardware()` | `() -> HardwareProfile` | Platform, architecture, CPU model string, total memory in GB, and whether the machine is Apple Silicon. | | ||
| | `all_engines()` | `() -> List[EngineAdapter]` | An adapter instance for every supported engine (`omlx`, `llama.cpp`). | | ||
| | `resolve_engines(names)` | `(names: List[str]) -> List[EngineAdapter]` | Adapters for a deduped, user-supplied engine list; raises on an unrecognized name. | | ||
| | `benchmark_engine(adapter, *, model, ...)` | `(adapter, *, model: str, max_tokens=None, prompts=None, verbose=False, on_progress=None) -> EngineBenchmarkResult` | Runs the fixed prompt sweep against one engine and returns a structured result. Never raises for "engine not installed" or one failed prompt -- that state lives in the returned object. | | ||
| | `recommend(results)` | `(results: List[EngineBenchmarkResult]) -> Optional[Recommendation]` | The engine with the highest measured average tok/s among installed, successfully tested engines. | | ||
| | `compare_to_cloud(cloud_model)` | `(cloud_model: str) -> Optional[CostComparison]` | A static, dated per-1K-output-token price for a known cloud model (currently `claude-5-haiku`, `claude-5-sonnet`) plus a disclosure note, or `None` for a model it doesn't recognize. | | ||
| | `report_to_dict(report)` / `write_json_report(report, path)` | `(report: BenchmarkReport) -> dict` / `(report, path: str) -> None` | Serialize a `BenchmarkReport` to the same camelCase JSON shape the CLI's `--json` / `--out` produce. | | ||
| ```python | ||
| from inferbench import benchmark_engine, detect_hardware, all_engines, recommend, compare_to_cloud | ||
| hardware = detect_hardware() | ||
| results = [ | ||
| benchmark_engine(adapter, model="qwen2.5-1.5b-instruct-4bit") | ||
| for adapter in all_engines() | ||
| ] | ||
| best = recommend(results) | ||
| print(f"{hardware.cpu_model}: {best.engine} -- {best.reason}") | ||
| # What would the same output volume cost on a cloud API instead? | ||
| cost = compare_to_cloud("claude-5-haiku") | ||
| if cost: | ||
| print(f"{cost.cloud_model}: ${cost.cloud_cost_per_1k_tokens_usd}/1K tokens (snapshot {cost.pricing_snapshot_date})") | ||
| ``` | ||
| ## How the measurement works | ||
@@ -120,17 +178,25 @@ | ||
| Instead, InferBench starts each engine's own already-standardized OpenAI-compatible HTTP server (`omlx serve`, `llama-server`) and sends the exact same prompts through the exact same measurement code to every engine, timing the full response (not just time-to-first-byte -- an earlier version of this code measured elapsed time right after `fetch()` resolved, which only captures HTTP headers arriving, not generation finishing, and produced a physically impossible 64,646 tok/s during a real end-to-end test run before the bug was caught and fixed). This is the only approach that is genuinely apples-to-apples across engines with fundamentally different internals, and the only one that works at all for `omlx`. | ||
| Instead, InferBench starts each engine's own already-standardized OpenAI-compatible HTTP server (`omlx serve`, `llama-server`) and sends the exact same prompts through the exact same measurement code to every engine, timing the full response (not just time-to-first-byte). This is the only approach that is genuinely apples-to-apples across engines with fundamentally different internals, and the only one that works at all for `omlx`. | ||
| ## What "recommended" means (and doesn't) | ||
| **What "recommended" means (and doesn't):** the recommendation in every report names the engine with the highest measured average tokens/second **on this specific run, this specific hardware, this specific model** -- not a general claim about which engine is best. A different model, a different machine, or a different day's thermal conditions can change the answer; two runs during this tool's own development produced opposite rankings between `omlx` and `llama.cpp` on the same hardware and model, which is itself the reason this tool measures live rather than quoting a fixed number. | ||
| The recommendation in every report is scoped explicitly: it names the engine with the highest measured average tokens/second **on this specific run, this specific hardware, this specific model** -- not a general claim about which engine is best. A different model, a different machine, or a different day's thermal conditions can change the answer; two runs during this tool's own development produced opposite rankings between `omlx` and `llama.cpp` on the same hardware and model, which is itself the reason this tool measures live rather than quoting a fixed number. | ||
| ## Comparison | ||
| | | InferBench | A static "definitive 2026 guide" comparison article | | ||
| |---|---|---| | ||
| | Measures | Your own hardware, live | The author's machine, once | | ||
| | Reproducible by you | Yes -- rerun any time | No -- you cannot rerun someone else's blog post | | ||
| | Stays current as engines update | Yes | No -- frozen at publish date | | ||
| | Vendor-neutral | Yes -- no engine of its own | Varies by author | | ||
| Three real, independently maintained tools sit in the same space, each with a different scope. Any cell not pulled from the linked project's own docs is marked accordingly. | ||
| | | InferBench | [llama-bench](https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md) (bundled with llama.cpp) | [local-llm-bench](https://github.com/famstack-dev/local-llm-bench) | [inference-benchmarker](https://github.com/huggingface/inference-benchmarker) (Hugging Face) | | ||
| |---|---|---|---|---| | ||
| | Engines covered | omlx, llama.cpp | llama.cpp only | Ollama, LM Studio, omlx, any OpenAI-compatible endpoint | Any OpenAI-compatible chat API (TGI, vLLM, etc.) | | ||
| | What it measures | Single-request avg/min/max tok/s across a fixed 8-prompt sweep | Prompt-processing and token-generation tok/s with tunable batch size, cache type, thread count | "Effective" tok/s (output tokens / total wall-clock including prefill) across custom real-world scenarios | Concurrency/throughput sweep at increasing request rates (QPS), production-serving focused | | ||
| | Cross-engine in one run | Yes | No -- one engine only | Yes, engine chosen per invocation | Yes, any server with the API, per invocation | | ||
| | Output formats | Human table, JSON | Markdown, CSV, JSON, JSONL, SQL | JSON to disk + a separate `compare.py` script | JSON | | ||
| | Distribution | npm + PyPI, `pip install` / `npm install -g` | Ships inside the llama.cpp build, no separate package | `git clone` + `python3 bench.py` (no PyPI/npm package) | `cargo install`, prebuilt binary, or Docker image | | ||
| | Platform | Cross-platform for llama.cpp; omlx is Apple Silicon-only | Cross-platform (same as llama.cpp) | Documented and demonstrated for Apple Silicon (MLX/GGUF engines) | Cross-platform, built for GPU server deployments | | ||
| ## Why this exists | ||
| Local inference on consumer hardware is now the default path for a growing share of developers, and every engine's own comparison against its competitors has an obvious incentive problem: no vendor is a disinterested judge of its own numbers. InferBench has no engine of its own to sell, which is the entire point. | ||
| The harder question this tool actually answers isn't "which engine is fastest in general" -- there is no such answer, because it depends on your exact hardware, your exact model, and your exact workload. It's "which engine is fastest **right now, on this machine, for this model**" -- a question only a tool that runs on your own hardware can answer honestly. | ||
| ## Documentation | ||
@@ -142,8 +208,2 @@ | ||
| ## Demo | ||
| Install, first run, and a real omlx benchmark against a cached model: | ||
|  | ||
| Machine-readable output written to a file with `--json --out`, useful for CI or for an agent parsing the result: | ||
@@ -159,3 +219,3 @@ | ||
| **How is InferBench different from llama.cpp's own `llama-bench`?** | ||
| `llama-bench` (bundled with llama.cpp) only benchmarks llama.cpp itself, with fine-grained tuning knobs (batch size, cache type, thread count, and more). InferBench benchmarks *across* engines -- currently `omlx` and `llama.cpp` -- using the same prompt set and the same measurement code for both, so the resulting tokens/second numbers are directly comparable to each other on your hardware, not just tunable in isolation for one engine. | ||
| `llama-bench` (bundled with llama.cpp) only benchmarks llama.cpp itself, with fine-grained tuning knobs (batch size, cache type, thread count, repetitions, and more) and outputs to Markdown, CSV, JSON, JSONL, or SQL. InferBench benchmarks *across* engines -- currently `omlx` and `llama.cpp` -- using the same prompt set and the same measurement code for both, so the resulting tokens/second numbers are directly comparable to each other on your hardware, not just tunable in isolation for one engine. | ||
@@ -162,0 +222,0 @@ **Does InferBench work on Linux and Windows, or only macOS?** |
New alerts
URL strings
Supply chain riskPackage contains fragments of external URLs or IP addresses, which the package may be accessing at runtime.
Fixed alerts
URL strings
Supply chain riskPackage contains fragments of external URLs or IP addresses, which the package may be accessing at runtime.
Improved metrics
- Total package byte prevSize
65404
11.03%- Number of lines in readme file
250
31.58%No dependency changes