evalsense

evalsense

JS-native LLM evaluation framework with Jest-like API and statistical assertions

evalsense brings classical ML-style statistical evaluation to LLM systems in JavaScript. Instead of evaluating individual test cases, evalsense evaluates entire datasets and computes confusion matrices, precision/recall, F1 scores, and other statistical metrics.

New in v0.3.2: Enhanced assertion reporting - all assertions (passed and failed) now display expected vs actual values, and chained assertions evaluate completely instead of short-circuiting on first failure! New in v0.3.0: Regression assertions (MAE, RMSE, R²) and flexible ID matching for custom identifier fields! See migration guide. New in v0.2.x: Built-in adapters for OpenAI, Anthropic, and OpenRouter - no boilerplate needed! New in v0.2.0: LLM-powered metrics for hallucination, relevance, faithfulness, and toxicity detection. See migration guide.

Why evalsense?

Most LLM evaluation tools stop at producing scores (accuracy, relevance, hallucination). evalsense goes further by:

• ✅ Computing confusion matrices to reveal systematic failure patterns
• ✅ Analyzing false positives vs false negatives across datasets
• ✅ Treating metrics as predictions, not truth (and validating them statistically)
• ✅ Providing a Jest-like API that fits naturally into JS/Node workflows
• ✅ Supporting deterministic CI/CD integration with specific exit codes

Features

• 📊 Dataset-level evaluation - evaluate distributions, not single examples
• 🎯 Statistical rigor - confusion matrices, precision/recall, F1, regression metrics
• 🧪 Jest-like API - familiar describe() and test patterns
• 🤖 LLM-powered metrics - hallucination, relevance, faithfulness, toxicity with explainable reasoning
• ⚡ Dual evaluation modes - choose between accuracy (per-row) or cost efficiency (batch)
• 🔄 CI-friendly - deterministic execution, machine-readable reports
• 🚀 JS-native - first-class TypeScript support, works with any Node.js LLM library
• 🔌 Composable - evaluate outputs from your existing LLM code

Installation

npm install --save-dev evalsense

Or with yarn:

yarn add -D evalsense

Quick Start

Create a file named sentiment.eval.js:

import { describe, evalTest, expectStats } from "evalsense";
import { readFileSync } from "fs";

// Your model function - can be any JS function
function classifySentiment(text) {
  const lower = text.toLowerCase();
  const hasPositive = /love|amazing|great|fantastic|perfect/.test(lower);
  const hasNegative = /terrible|worst|disappointed|waste/.test(lower);
  return hasPositive && !hasNegative ? "positive" : "negative";
}

describe("Sentiment classifier", () => {
  evalTest("accuracy above 80%", async () => {
    // 1. Load ground truth data
    const groundTruth = JSON.parse(readFileSync("./sentiment.json", "utf-8"));

    // 2. Run your model and collect predictions
    const predictions = groundTruth.map((record) => ({
      id: record.id,
      sentiment: classifySentiment(record.text),
    }));

    // 3. Assert on statistical properties
    expectStats(predictions, groundTruth)
      .field("sentiment")
      .toHaveAccuracyAbove(0.8)
      .toHaveRecallAbove("positive", 0.7)
      .toHavePrecisionAbove("positive", 0.7)
      .toHaveConfusionMatrix();
  });
});

Create sentiment.json:

[
  { "id": "1", "text": "I love this product!", "sentiment": "positive" },
  { "id": "2", "text": "Terrible experience.", "sentiment": "negative" },
  { "id": "3", "text": "Great quality!", "sentiment": "positive" }
]

Run the evaluation:

npx evalsense run sentiment.eval.js

Usage

Basic Classification Example

import { describe, evalTest, expectStats } from "evalsense";
import { readFileSync } from "fs";

describe("Spam classifier", () => {
  evalTest("high precision and recall", async () => {
    const groundTruth = JSON.parse(readFileSync("./emails.json", "utf-8"));

    const predictions = groundTruth.map((record) => ({
      id: record.id,
      isSpam: classifyEmail(record.text),
    }));

    expectStats(predictions, groundTruth)
      .field("isSpam")
      .toHaveAccuracyAbove(0.9)
      .toHavePrecisionAbove(true, 0.85) // Precision for spam=true
      .toHaveRecallAbove(true, 0.85) // Recall for spam=true
      .toHaveConfusionMatrix();
  });
});

Continuous Scores with Binarization

import { describe, evalTest, expectStats } from "evalsense";
import { readFileSync } from "fs";

describe("Hallucination detector", () => {
  evalTest("detect hallucinations with 70% recall", async () => {
    const groundTruth = JSON.parse(readFileSync("./outputs.json", "utf-8"));

    // Your model returns a continuous score (0.0 to 1.0)
    const predictions = groundTruth.map((record) => ({
      id: record.id,
      hallucinated: computeHallucinationScore(record.output),
    }));

    // Binarize the score at threshold 0.3
    expectStats(predictions, groundTruth)
      .field("hallucinated")
      .binarize(0.3) // >= 0.3 means hallucinated
      .toHaveRecallAbove(true, 0.7)
      .toHavePrecisionAbove(true, 0.6)
      .toHaveConfusionMatrix();
  });
});

Multi-class Classification

import { describe, evalTest, expectStats } from "evalsense";
import { readFileSync } from "fs";

describe("Intent classifier", () => {
  evalTest("balanced performance across intents", async () => {
    const groundTruth = JSON.parse(readFileSync("./intents.json", "utf-8"));

    const predictions = groundTruth.map((record) => ({
      id: record.id,
      intent: classifyIntent(record.query),
    }));

    expectStats(predictions, groundTruth)
      .field("intent")
      .toHaveAccuracyAbove(0.85)
      .toHaveRecallAbove("purchase", 0.8)
      .toHaveRecallAbove("support", 0.8)
      .toHaveRecallAbove("general", 0.7)
      .toHaveConfusionMatrix();
  });
});

Parallel Model Execution with LLMs

For LLM calls or slow operations, use Promise.all with chunking for concurrency control:

import { describe, evalTest, expectStats } from "evalsense";
import { readFileSync } from "fs";

// Helper for parallel execution with concurrency limit
async function mapConcurrent(items, fn, concurrency = 5) {
  const results = [];
  for (let i = 0; i < items.length; i += concurrency) {
    const chunk = items.slice(i, i + concurrency);
    results.push(...(await Promise.all(chunk.map(fn))));
  }
  return results;
}

describe("LLM classifier", () => {
  evalTest("classification accuracy", async () => {
    const groundTruth = JSON.parse(readFileSync("./data.json", "utf-8"));

    // Run with concurrency=5
    const predictions = await mapConcurrent(
      groundTruth,
      async (record) => {
        const response = await callLLM(record.text);
        return { id: record.id, category: response.category };
      },
      5
    );

    expectStats(predictions, groundTruth).field("category").toHaveAccuracyAbove(0.9);
  });
});

Test Lifecycle Hooks

import { describe, evalTest, beforeAll, afterAll, beforeEach, afterEach } from "evalsense";

describe("Model evaluation", () => {
  let model;

  beforeAll(async () => {
    // Load model once before all tests
    model = await loadModel();
  });

  afterAll(async () => {
    // Cleanup after all tests
    await model.dispose();
  });

  beforeEach(() => {
    // Reset state before each test
    model.reset();
  });

  afterEach(() => {
    // Cleanup after each test
    console.log("Test completed");
  });

  evalTest("test 1", async () => {
    // ...
  });

  evalTest("test 2", async () => {
    // ...
  });
});

CLI Usage

Run Evaluations

# Run all eval files in current directory
npx evalsense run

# Run specific file or directory
npx evalsense run tests/eval/

# Filter tests by name
npx evalsense run --filter "accuracy"

# Output JSON report
npx evalsense run --output report.json

# Use different reporters
npx evalsense run --reporter console  # default
npx evalsense run --reporter json
npx evalsense run --reporter both

# Bail on first failure
npx evalsense run --bail

# Set timeout (in milliseconds)
npx evalsense run --timeout 60000

List Eval Files

# List all discovered eval files
npx evalsense list

# List files in specific directory
npx evalsense list tests/

API Reference

Core API

describe(name, fn)

Groups related evaluation tests (like Jest's describe).

describe("My model", () => {
  // eval tests go here
});

evalTest(name, fn) / test(name, fn) / it(name, fn)

Defines an evaluation test.

evalTest("should have 90% accuracy", async () => {
  // test implementation
});

Dataset Loading

evalsense doesn't dictate how you load data or run your model. Use standard Node.js tools:

import { readFileSync } from "fs";

// Load ground truth
const groundTruth = JSON.parse(readFileSync("./data.json", "utf-8"));

// Run your model however you want
const predictions = groundTruth.map(runYourModel);

// Or use async operations
const predictions = await Promise.all(
  groundTruth.map(async (item) => {
    const result = await callLLM(item.text);
    return { id: item.id, prediction: result };
  })
);

Helper functions available (optional):

• loadDataset(path) - Simple JSON file loader
• runModel(dataset, fn) - Sequential model execution
• runModelParallel(dataset, fn, concurrency) - Parallel execution with concurrency limit

Assertions

expectStats(predictions, groundTruth)

Creates a statistical assertion chain from predictions and ground truth. Aligns by id field.

expectStats(predictions, groundTruth)
  .field("prediction")
  .toHaveAccuracyAbove(0.8)
  .toHaveF1Above(0.75)
  .toHaveConfusionMatrix();

New in v0.3.2: Enhanced Assertion Reporting

• All assertions (passed and failed) now display expected vs actual values
• Chained assertions evaluate completely instead of short-circuiting on first failure
• See all metric results in a single run for better debugging

One-argument form (distribution assertions only):

// For distribution monitoring without ground truth
expectStats(predictions).field("confidence").toHavePercentageAbove(0.7, 0.8);

Common use cases:

• Classification evaluation with ground truth
• Regression evaluation (MAE, RMSE, R²)
• Validating LLM judges against human labels
• Distribution monitoring without ground truth

Field Selection

.field(fieldName)

Selects a field for evaluation.

expectStats(result).field("sentiment");

.binarize(threshold)

Converts continuous scores to binary (>=threshold is true).

expectStats(result)
  .field("score")
  .binarize(0.5) // score >= 0.5 is true
  .toHaveAccuracyAbove(0.8);

Available Assertions

Classification Metrics

// Accuracy
.toHaveAccuracyAbove(threshold)
.toHaveAccuracyBelow(threshold)
.toHaveAccuracyBetween(min, max)

// Precision (per class)
.toHavePrecisionAbove(className, threshold)
.toHavePrecisionBelow(className, threshold)

// Recall (per class)
.toHaveRecallAbove(className, threshold)
.toHaveRecallBelow(className, threshold)

// F1 Score
.toHaveF1Above(threshold)           // Overall F1
.toHaveF1Above(className, threshold) // Per-class F1

// Confusion Matrix
.toHaveConfusionMatrix()  // Prints confusion matrix

Distribution Assertions (Pattern 1)

Distribution assertions validate output distributions without requiring ground truth. Use these to monitor that model outputs stay within expected ranges.

// Assert that at least 80% of confidence scores are above 0.7
expectStats(predictions).field("confidence").toHavePercentageAbove(0.7, 0.8);

// Assert that at least 90% of toxicity scores are below 0.3
expectStats(predictions).field("toxicity").toHavePercentageBelow(0.3, 0.9);

// Chain multiple distribution assertions
expectStats(predictions)
  .field("score")
  .toHavePercentageAbove(0.5, 0.6) // At least 60% above 0.5
  .toHavePercentageBelow(0.9, 0.8); // At least 80% below 0.9

Use cases:

• Monitor confidence score distributions
• Validate schema compliance rates
• Check output range constraints
• Ensure score distributions remain stable over time

See Distribution Assertions Example for complete examples.

Judge Validation (Pattern 1b)

Validate judge outputs against human-labeled ground truth using the two-argument expectStats API:

// Judge outputs (predictions from your judge/metric)
const judgeOutputs = [
  { id: "1", hallucinated: true },
  { id: "2", hallucinated: false },
  { id: "3", hallucinated: true },
];

// Human labels (ground truth)
const humanLabels = [
  { id: "1", hallucinated: true },
  { id: "2", hallucinated: false },
  { id: "3", hallucinated: false },
];

// Validate judge performance
expectStats(judgeOutputs, humanLabels)
  .field("hallucinated")
  .toHaveRecallAbove(true, 0.9) // Don't miss hallucinations
  .toHavePrecisionAbove(true, 0.7) // Some false positives OK
  .toHaveConfusionMatrix();

Use cases:

• Evaluate LLM-as-judge accuracy
• Validate heuristic metrics against human labels
• Test automated detection systems (refusal, policy compliance)
• Calibrate metric thresholds

Two-argument expectStats:

expectStats(actual, expected).field("fieldName").toHaveAccuracyAbove(0.8);

The first argument is your predictions (judge outputs), the second is ground truth (human labels). Both must have matching id fields for alignment.

See Judge Validation Example for complete examples.

For comprehensive guidance on evaluating agent systems, see Agent Judges Design Patterns.

Dataset Format

Datasets must be JSON arrays where each record has an id or _id field:

[
  {
    "id": "1",
    "text": "input text",
    "label": "expected_output"
  },
  {
    "id": "2",
    "text": "another input",
    "label": "another_output"
  }
]

Requirements:

• Each record MUST have id or _id for alignment
• Ground truth fields (e.g., label, sentiment, category) are compared against model outputs
• Model functions must return predictions with matching id

Exit Codes

evalsense returns specific exit codes for CI integration:

• 0 - Success (all tests passed)
• 1 - Assertion failure (statistical thresholds not met)
• 2 - Integrity failure (dataset alignment issues)
• 3 - Execution error (test threw exception)
• 4 - Configuration error (invalid CLI options)

Writing Eval Files

Eval files use the .eval.js or .eval.ts extension and are discovered automatically:

project/
├── tests/
│   ├── classifier.eval.js
│   └── hallucination.eval.js
├── data/
│   └── dataset.json
└── package.json

Run with:

npx evalsense run tests/

Examples

See the examples/ directory for complete examples:

• classification.eval.js - Binary sentiment classification
• hallucination.eval.js - Continuous score binarization
• distribution-assertions.eval.js - Distribution monitoring without ground truth
• judge-validation.eval.js - Validating judges against human labels

Field Types

evalsense automatically determines evaluation metrics based on field values:

• Boolean (true/false) → Binary classification metrics
• Categorical (strings) → Multi-class classification metrics
• Numeric (numbers) → Regression metrics (MAE, MSE, RMSE, R²)
• Numeric + threshold → Binarized classification metrics

LLM-Based Metrics (v0.2.0+)

evalsense includes LLM-powered metrics for hallucination detection, relevance assessment, faithfulness verification, and toxicity detection.

Quick Setup

import { setLLMClient, createOpenAIAdapter } from "evalsense/metrics";
import { hallucination, relevance, faithfulness, toxicity } from "evalsense/metrics/opinionated";

// 1. Configure your LLM client (one-time setup)
setLLMClient(
  createOpenAIAdapter(process.env.OPENAI_API_KEY, {
    model: "gpt-4-turbo-preview",
    temperature: 0,
  })
);

// 2. Use metrics in evaluations
const results = await hallucination({
  outputs: [{ id: "1", output: "Paris has 50 million people." }],
  context: ["Paris has approximately 2.1 million residents."],
});

console.log(results[0].score); // 0.9 (high hallucination)
console.log(results[0].reasoning); // "Output claims 50M, context says 2.1M"

Available Metrics

• hallucination() - Detects claims not supported by context
• relevance() - Measures query-response alignment
• faithfulness() - Verifies outputs don't contradict sources
• toxicity() - Identifies harmful or inappropriate content

Evaluation Modes

Choose between accuracy and cost:

// Per-row: Higher accuracy, higher cost (N API calls)
await hallucination({
  outputs,
  context,
  evaluationMode: "per-row", // default
});

// Batch: Lower cost, single API call
await hallucination({
  outputs,
  context,
  evaluationMode: "batch",
});

Built-in Provider Adapters

evalsense includes ready-to-use adapters for popular LLM providers:

OpenAI (GPT-4, GPT-3.5)

import { createOpenAIAdapter } from "evalsense/metrics";

// npm install openai
setLLMClient(
  createOpenAIAdapter(process.env.OPENAI_API_KEY, {
    model: "gpt-4-turbo-preview", // or "gpt-3.5-turbo" for lower cost
    temperature: 0,
    maxTokens: 4096,
  })
);

Anthropic (Claude)

import { createAnthropicAdapter } from "evalsense/metrics";

// npm install @anthropic-ai/sdk
setLLMClient(
  createAnthropicAdapter(process.env.ANTHROPIC_API_KEY, {
    model: "claude-3-5-sonnet-20241022", // or "claude-3-haiku-20240307" for speed
    maxTokens: 4096,
  })
);

OpenRouter (100+ models from one API)

import { createOpenRouterAdapter } from "evalsense/metrics";

// No SDK needed - uses fetch
setLLMClient(
  createOpenRouterAdapter(process.env.OPENROUTER_API_KEY, {
    model: "anthropic/claude-3.5-sonnet", // or "openai/gpt-3.5-turbo", etc.
    temperature: 0,
    appName: "my-eval-system",
  })
);

Custom Adapter (for any provider)

setLLMClient({
  async complete(prompt) {
    // Implement for your LLM provider
    const response = await yourLLM.generate(prompt);
    return response.text;
  },
});

Learn More

• LLM Metrics Guide - Complete usage guide
• LLM Adapters Guide - Implement adapters for different providers
• Migration Guide - Upgrade from v0.1.x
• Examples - Working code examples

Philosophy

evalsense is built on the principle that metrics are predictions, not facts.

Instead of treating LLM-as-judge metrics (relevance, hallucination, etc.) as ground truth, evalsense:

• Treats them as weak labels from a model
• Validates them statistically against human references when available
• Computes confusion matrices to reveal bias and systematic errors
• Focuses on dataset-level distributions, not individual examples

Contributing

Contributions are welcome! Please see CLAUDE.md for development guidelines.

License

MIT © Mohit Joshi

Made with ❤️ for the JS/Node.js AI community