evalite

What Is Evalite?

• Fully open source: No API Key required.
• Local-first: runs on your machine, your data never leaves your laptop.
• Based on Vitest, the best TypeScript test runner around.
• Terminal UI for quick prototyping.
• Supports tracing and custom scorers.

How Do I Learn More?

• Read the Docs
• Join the Discord

Evalite Is Experimental

Evalite is still an experimental project. I'm actively working on it, and for now am pushing breaking changes.

If you run into any unexpected behavior:

• Delete the node_modules/.evalite folder.
• Update evalite to the latest version.
• Rerun your evals.

If, after that, you run into unexpected behavior, report an issue.

Guides

Watch Mode

You can run Evalite in watch mode by running evalite watch:

evalite watch

This will watch for changes to your .eval.ts files and re-run the evals when they change.

[!IMPORTANT]

I strongly recommend implementing a caching layer in your LLM calls when using watch mode. This will keep your evals running fast and avoid burning through your API credits.

Running Specific Files

You can run specific files by passing them as arguments:

evalite my-eval.eval.ts

This also works for watch mode:

evalite watch my-eval.eval.ts