Python SDK for reporting benchmark results to the Benchmark Command Center
Local-first evals for agents, not just prompts.
A framework for evaluating large multi-modality language models
Evaluation infrastructure for GUI agent benchmarks
In-loop evaluation tasks for language modeling
The integrity layer for your evals: fail-closed release gates with signed, tamper-evident, pre-registered receipts — for pytest, promptfoo, claude plugin eval, or any scores. Runs in any CI, no AI agent required.
Pytest-style framework for evaluating Model Context Protocol (MCP) servers.
Evaluation for the Pure AB-3D-MOT.
AgentLab — a modular framework for building, evaluating, and comparing LLM-powered agents.
EvalFloor: is your LLM eval improvement real? Trying k prompt variants and keeping the best scores points on noise alone -- this computes how many, so you can tell a real gain from a lucky sample.
Lifelong learning Speaker diarization baseline system.
Unlearning Benchmark for Text-to-Image Models
A boring, config-driven harness for evaluating AI systems.
Evaluation of LbAgents workflows on replayed real failures
An open benchmark format for LLM evaluation, with native agentic support.
Open-source library of metrics
A research toolkit for measuring and analyzing model behavior.
crit-eval: one speed, quality and exactness suite that every inference engine runs the same way
The public evals of the Positronic platform: their definitions, labelers, scoring rules and episode lists
An evaluation framework for AI agents. Bring any benchmark, agent or capability, run it on your machines, and inspect the evidence behind every score. Bring anything, run it, prove it.
Audits LLM eval datasets for the flaws that make evaluations silently lie.
A framework for evaluating language models - packaged by NVIDIA
The vocabulary an attended rollout records: the operator's verdict and the progress ladder
Library for RAG evaluation
Evaluate habitat classifiers on Tessera satellite embeddings
synthetic evals for agents
Unlearning Benchmark for Text-to-Image Models
LLM-as-a-judge scoring for AI systems, via provider batch APIs
Regression testing for LLM agents
Send eval runs and production feedback to Evalban, the board that knows when your model got worse.
Python SDK for evaluating LLM Applications
A local HTTP service for sandboxed multi-language code execution (Python, JS, Java, Rust) with auto-provisioning toolchains
Build and validate eval-run provenance manifests with deterministic directory digests.
Know what you are not testing before you ship - an eval coverage auditor for RAG and agent systems.
Regulated LLM Eval Framework — evaluation for LLMs in regulated domains (tax, accounting, legal) with 152-FZ compliance
Verdict eval engine: binary judges, structural checks, semantic drift, and clustering
Closed-loop eval monitoring for LLM-powered products
Run rubric-spec adapter conformance checks and render deterministic status badges.
Turn production agent failures into a small, reviewed regression suite.