Simple benchmark framework (in active development)
A ``pytest`` fixture for benchmarking code. It will group the tests into rounds that are calibrated to the chosen timer.
Lightweight grading toolkit for environment-based tasks.
A Python library for anomaly detection across tabular, time series, graph, text, image, and audio data. 61 detectors, benchmark-backed ADEngine orchestration, and an agentic workflow for AI agents.
Pytest plugin to create CodSpeed benchmarks
Massive Text Embedding Benchmark
Store data created during your pytest tests execution, and retrieve it at the end of the session, e.g. for applicative benchmarking purposes.
Python module to run and analyze benchmarks
robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
provides a common interface to many IR ad-hoc ranking benchmarks, training datasets, etc.
Core Python benchmark code for ASV
Airspeed Velocity: A simple Python history benchmarking tool
Android SoC benchmarking CLI for latency, loaded latency, and memory subsystem characterization
The seismological machine learning benchmark collection
Reversible Data Transforms
A verified version of the WebArena Benchmark
One-click accuracy evaluation harness for SGLang.
Pytest benchmarking plugin for codeflash.ai - automatic code performance optimization
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
Run computer-use benchmark datasets on sandboxes
Benchmarking optimization solvers
OpenMMLab Detection Toolbox and Benchmark
Openpyxl-compatible Excel library with a Rust backend: speed up openpyxl code with one import change and edit .xlsx templates without losing formatting, conditional formatting, or data validation. WolfXL 2.0.21 measured 6.7-12.0x faster than openpyxl 3.1.5 on the committed read/write benchmark (small-file gains vary). MIT.
Python library for reading and writing VRP instances.
Benchmarking QRC measures the ability to store information of
SemiAnalysis AgentX inference benchmark harness, built on AIPerf
ML models + benchmark for tabular data classification and regression
A schema and toolkit for curating tabular datasets and benchmarking tasks (the data layer behind TabArena).
Metrics for multiple object tracker benchmarking.
Benchmark evaluation for tabular ML: Elo, win-rate, rank and improvability leaderboards (the engine behind TabArena)
Benchmark Runner Tool
A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents
FitBenchmarking: A tool for comparing fitting software
RelBench: Relational Deep Learning Benchmark
Cargo extension to build & run your codspeed benchmarks
OGBench: Benchmarking Offline Goal-Conditioned RL
A Python Toolbox for Benchmarking Machine Learning on Partially-Observed Time Series
ManiSkill3: A Unified Benchmark for Generalizable Manipulation Skills
NeMo Evaluator — benchmark environments, pluggable solvers, interceptor proxy, and decision-grade scoring for LLMs
Datazets is a python package to import well known example data sets.
Access the datasets, models and Remote Lab from Causal Chamber™.
Benchmarking multimodal tabular learning: tables whose columns include text or images
A command-line benchmarking tool
ParseBench: a benchmark and evaluation harness for document parsing systems on real-world enterprise documents