@empiricalrun/cli

empirical.run CLI

Empirical is the fastest way to test different LLMs, prompts and other model configurations, across all the scenarios that matter for your application.

With Empirical, you can:

• Run your test datasets locally against off-the-shelf models
• Test your own custom models and RAG applications (see how-to)
• Reports to view, compare, analyze outputs on a web UI
• Score your outputs with scoring functions
• Run tests on CI/CD (coming soon!)

Watch demo video | See all docs

Usage

Empirical bundles together a CLI and a web app. The CLI handles running tests and the web app visualizes results.

Everything runs locally, with a JSON configuration file, empiricalrc.json.

Required: Node.js 20+ needs to be installed on your system.

Start with a basic example

This example converts incoming unstructured user messages into structured JSON objects using an LLM.

1. Use the CLI to create a sample configuration file called empiricalrc.json.

   npx @empiricalrun/cli init
   

2. Run the test samples against the models with the run command.

   npx @empiricalrun/cli run
   

3. Use the ui command to open the reporter web app and see side-by-side results.

   npx @empiricalrun/cli ui
   

Make it yours

Edit the empiricalrc.json file to make Empirical work for your use-case.

• Configure which models to use
• Configure your test dataset
• Configure scoring functions to grade output quality