Turn production agent failures into a small, reviewed regression suite.
Export rubric-spec rubrics and normalize evaluation result shapes across common eval frameworks.
Python client for sending LLM traces to EvalWatch
A simple, safe single expression evaluator library.
Test your Python app's AI calls with evals in a short bash file.
A comprehensive evaluation toolkit for assessing Retrieval-Augmented Generation (RAG) outputs using linguistic, semantic, and fairness metrics
Wrapper around ast.literal_eval with new {foo='bar', key=None} dict syntax.
Decide whether an agent is good enough to ship. Honest statistics for pilot-scale case sets.
Small LLM eval and tracing harness: run any pipeline over a test set, trace tokens/latency/cost per case, score it, and gate CI on quality drops.
Decide whether an eval delta is a real regression or sampling noise.
Eval LLM
Decision-grade statistics for AI evals: paired comparisons, cluster-aware uncertainty, and power analysis on top of existing eval frameworks.
A flight simulator for AI agents - deterministic, stateful simulated worlds for agent evaluation.
Statistical significance & confidence intervals for LLM evals
CI/CD Quality Gate & Automated Red-Teaming for GenAI & LLM Applications
Local-First RAG Retrieval, Chunking & pgvector Diagnostic Workbench (CLI)
Security assessment framework for AI agents — adversarial test runner + server-side audit + scoring
Evaluation framework for Letta AI agents
PICAI Evaluation
Open evaluation kit for LLM systems
Evaluate the Lyapunov Spectrum of a Dynamical System
User-friendly evaluation framework: Eval Suite & Benchmarks: UHGEval, HaluEval, HalluQA, etc.
We help GenAI teams maintain high-accuracy for their Models in production.
A small, focused eval harness for evaluating AI systems in Python. Zero dependencies.
XBlocks to write short text and code entries with AI-driven evaluation
CLI tool that evaluates LLM outputs from production logs against a dual-dimension rubric.
Generate eval datasets from arbitrary sources
a fork of Faster-COCO-Eval modified specifically for the AI-TOD dataset
Read the ABSCAL.HIS file of RPG microwave radiometers (e.g. HATPRO) and visualize the history of the absolute calibration with liquid nitrogen.
A CI gate for LLM agents: error-analysis-first evals, a calibrated LLM judge, and judge-drift detection.
Evaluate large-language models for undesirable behaviors such as bias.
Evaluate product search quality with IR metrics, multi-judge LLM labeling, cost tracking, and CI quality gates.
A new package under development
One eval harness, many LLMs. Start poking. An agentic NetOps benchmark for local and frontier LLMs.
Detect flaky LLM eval cases across repeated runs. Pass-rate + standard-deviation per case, with per-case severity. Python port of @mukundakatta/eval-flake-detector.
The missing bridge between notebook eval and production A/B testing
Open-Source Evaluation for GenAI Applications.
Independent evaluation framework, benchmark suite, and dataset pipeline for Indonesian NLP text normalization and preprocessing.
The fastest, most cost-efficient LLM evaluation framework — 100+ metrics, parallel async eval, batched judging, native pytest integration.
Generate balanced AI eval fixtures from source examples, bugs, docs, and policies. Python port of @mukundakatta/eval-dataset-smith.
A small eval harness that tells you which cases your prompt change broke.
Typed, provider-independent micro-evaluations for Python
OpenCompass VLM Evaluation Kit for Eval-Scope
HiDevs quest eval client. Runs the local agent and uploads replies for scoring.
A professional multi-turn conversational AI agent evaluation framework supporting English, Arabic, and French.