Prove your graders: a grader-validation and benchmark-audit toolkit for agent builders — validate graders against labelled data and audit static prompt-to-gold suites for broken tests.
Evaluation toolkit for depth, RGB, ray and 3D point-map predictions
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation.
Evaluate LLM-backed products: campaigns over levers, judged and code-graded trials, cost and latency accounting, and analysis reports
Runtime helpers for Inspect AI evaluations: dataset loading with retries, sample ids, prompts and README generation
Serialization based on ast.literal_eval
Framework for evaluating agent skills across OpenCode, Claude Code, and Codex
How many eval examples do I need? Sample size and power planning for LLM evals
A package that lets you safely evaluate arithmetic expressions.
Pydantic schemas and a writer for the evals-viewer on-disk format — the Python writer side of the evals-viewer framework.
CAFA evaluator code
Evaluation toolbox for Sound Event Detection
Tool for safe (or less safe) evaluation of strings as math expressions
A library containing retrieval related components in Gen AI applications.
Open-source AI evaluation framework for LLM agents, prompts, and structured outputs
A powerful Quantum Photonic Framework
Agent evals and guardrails that run in one request, built on System One models (Jev, Kev, Laya).
Tracking evaluation metrics including ClearMOT and HOTA.
Prime Intellect Traces SDK - Upload, query and export traces
Shared eval tools for single-cell bench, spatial bench, and future biology benchmarks.
Core ARL data model library
Evaluating and scoring financial data
Python implementation of muParser math syntax parser with NumPy and xarray support
Prime Intellect Runs SDK - Track eval and training runs on the Prime platform
Evaluation infrastructure for self-evolving agents on the Harbor task standard: autoresearch and task streams, tamper-proof scoring, budgets on four axes, and resumable sweeps.
Guardrails and evals for AI agents: a policy gate for tool calls, a checker for invented facts, and a benchmark for shell-command gates. Deterministic, no API keys.
A pytest plugin for running and analyzing LLM evaluation tests
DataEval provides a simple interface to characterize image data and its impact on model performance across classification and object-detection tasks
Turn real LLM app logs into an eval suite and measure which LLM judges you can trust.
Math expression evaluator with exact Decimal arithmetic, gas metering and nesting-depth limits — powered by Rust
AlpacaEval : An Automatic Evaluator of Instruction-following Models
Lightweight aspect-based evaluation framework with YAML check definitions.
ranx: A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion
Standalone framework to evaluate agent correctness based on portable OpenTelemetry traces
Query metadata from sdists / bdists / installed packages. Safer fork of pkginfo to avoid doing arbitrary imports and eval.
Judges agent/RAG outputs (relevance, hallucination, accuracy, routing, prompt injection) using Laya instead of LLM-as-judge -- usable as an MCP server or as a plain Python library.
An auditor for LLM evaluations — tells you whether you can trust your eval results.
Static checks for Inspect AI evaluations: structure, tests, best practices and sandbox pinning
Async red-teaming kernel for LLM applications: vulnerability catalog, attack techniques, multi-turn jailbreak algorithms, judges and safety frameworks
LLM eval framework with first-class token-cost tracking. Did your prompt change save money without losing quality?
eval-mm is a tool for evaluating Multi-Modal Large Language Models.
Evaluation metrics for single-cell perturbation predictions
pytest plugin with shared test gates, fixtures and helpers for Inspect AI evaluations
Evaluate the GoodData AI agent against your own questions and models.
The AID Framework Python runtime: model gateway with pinning and cost budgets, prompt registry, tool executor, OTel GenAI tracing, plain-async pipeline, and an eval harness that gates on statistical regression against a stored baseline.