promptloop

PromptLoop

Autonomous prompt optimizer. Give it a prompt and test cases — it will iteratively mutate, score, and improve your prompt using the Karpathy autoresearch pattern.

Try the live demo at promptloop.vercel.app

Before / After

	Score
Original prompt (hand-written)	55 / 100
Optimized prompt (15 iterations, $1.40)	82 / 100

PromptLoop vs. Alternatives

Feature	PromptLoop	DSPy	PromptFoo	OpenAI Optimizer
Optimizes individual prompts	Yes	No (LLM programs)	No (testing only)	Yes
TypeScript-native	Yes	No (Python)	Yes	No (Python)
Works with any model	Yes	Yes	Yes	OpenAI only
CLI + Web UI	Yes	CLI only	CLI + UI	API only
Mutation strategies	6 strategies	Bootstrapping	N/A	Gradient-free
Real-time progress	SSE streaming	No	No	No
Self-hostable	Yes	Yes	Yes	No

promptloop init linkedin-hooks
promptloop run

How It Works

Load prompt.md + test-cases.json
         │
    ┌────▼────┐
    │ Baseline │  Execute prompt against all test cases, score with LLM judge
    └────┬────┘
         │
    ┌────▼──────────────────────┐
    │ Loop (N iterations)       │
    │                           │
    │  1. Failure report        │  Identify lowest-scoring test cases
    │  2. Mutate                │  Optimizer agent picks a strategy
    │  3. Re-score              │  Execute mutated prompt, judge outputs
    │  4. Keep or revert        │  Score improved? Keep. Otherwise revert.
    │  5. Log to history.jsonl  │
    │                           │
    └────┬──────────────────────┘
         │
    ┌────▼────┐
    │ Output  │  Optimized prompt.md + progress.png + report.json
    └─────────┘

Quick Start

Install

npm install -g promptloop

Create a project

promptloop init linkedin-hooks
cd linkedin-hooks

This creates:

• prompt.md — the prompt to optimize
• test-cases.json — test cases with inputs and expected outputs
• promptloop.config.ts — model selection, budget, scoring criteria

Run optimization

export ANTHROPIC_API_KEY=sk-ant-...
promptloop run

Score without optimizing

promptloop eval

Configuration

// promptloop.config.ts
import type { PromptLoopConfig } from "promptloop";

const config: PromptLoopConfig = {
  // Model that executes your prompt
  targetModel: {
    provider: "anthropic",
    model: "claude-sonnet-4-5-20250929",
  },
  // Model that generates mutations
  optimizerModel: {
    provider: "anthropic",
    model: "claude-sonnet-4-5-20250929",
  },
  // Model that judges output quality
  judgeModel: {
    provider: "anthropic",
    model: "claude-sonnet-4-5-20250929",
  },
  maxIterations: 25,
  maxCostUsd: 5.0,
  parallelTestCases: 5,
  scoring: {
    mode: "llm-judge",
    criteria: "Score the output 0-100 on relevance, quality, and completeness.",
  },
  failureReportSize: 3,
  targetScore: 90,
};

export default config;

Models

Works with any model via two providers:

Provider	Models
anthropic	Claude Sonnet 4.5, Claude Opus 4.6, Claude Haiku 4.5
openai	GPT-4o, GPT-4.1, GPT-4.1-mini, GPT-4.1-nano

Use different models for different roles — e.g., cheap model as target, expensive model as judge.

Scoring Modes

• llm-judge — LLM scores each output against your criteria (default)
• programmatic — Your custom eval function scores outputs
• hybrid — Weighted combination of both

For programmatic scoring, export a function:

// eval.ts
import type { TestCase } from "promptloop";

export default function score(output: string, testCase: TestCase): number {
  // Return 0-100
  if (output.length > 200) return 30; // too long
  if (!output.includes(testCase.input.split(" ")[0])) return 50; // missed topic
  return 85;
};

Then set scoring.evalPath: "./eval.ts" in your config.

Mutation Strategies

The optimizer uses Claude tool use to apply structured mutations:

Strategy	When Used
sharpen	Tighten vague instructions
add_example	Model misunderstands format/tone
remove	Contradictory or redundant rules
restructure	Information is buried or poorly ordered
constrain	Output goes off-track
expand	Instructions are under-specified

One mutation per iteration. The optimizer learns from history — if a strategy was reverted, it tries a different approach.

CLI Reference

promptloop init [template]       # Scaffold a project (templates: linkedin-hooks, blank)
promptloop run [options]         # Run optimization loop
promptloop eval [options]        # Score current prompt (no optimization)
promptloop generate-tests [opt]  # Generate test cases from a prompt using AI
promptloop diff [options]        # Show diff between original and optimized prompt
promptloop estimate [options]    # Estimate cost for an optimization run

# Run options
  -p, --prompt <path>      # Prompt file (default: prompt.md)
  -t, --tests <path>       # Test cases file (default: test-cases.json)
  -c, --config <path>      # Config file (default: promptloop.config.ts)
  -o, --output <path>      # Output directory (default: .promptloop)
  -n, --iterations <n>     # Override max iterations
  --budget <usd>           # Override max cost

Programmatic API

import { run, scorePrompt, generateMutation } from "promptloop";

// Run the full optimization loop
const report = await run({
  promptPath: "./prompt.md",
  testCasesPath: "./test-cases.json",
  config: { /* ... */ },
  outputDir: "./.promptloop",
});

console.log(`Improved ${report.baselineScore} → ${report.finalScore}`);

Output Files

After a run, .promptloop/ contains:

• history.jsonl — every iteration as a JSON line (append-only, crash-safe)
• progress.png — score chart with baseline, improvements, and reverts
• report.json — full run summary

Badge

Add this badge to your project's README to show your prompts were optimized with PromptLoop:

[![Optimized with PromptLoop](https://img.shields.io/badge/prompts-optimized%20with%20PromptLoop-C2410C)](https://github.com/jerrysoer/promptloop)

Cost

Typical costs for a 25-iteration run with 10 test cases:

Configuration	Estimated Cost
Sonnet for all three roles	$1–3
Haiku target, Sonnet optimizer/judge	$0.50–1.50
Sonnet target, Opus optimizer, Sonnet judge	$3–8

Set maxCostUsd in config to cap spending. The loop stops when the budget is reached.

FAQ

How is this different from DSPy? DSPy optimizes LLM programs (chains of calls). PromptLoop optimizes individual prompts — simpler scope, TypeScript-native, works with any model.

How is this different from PromptFoo? PromptFoo tests prompts. PromptLoop optimizes them. They are complementary — use PromptFoo to evaluate, PromptLoop to improve.

Does it work with local models? Yes — set baseUrl in your model config to point to any OpenAI-compatible API (Ollama, vLLM, etc.).

Can I resume a failed run? Yes — use the "Continue Run" button in the web UI. The CLI supports resume via the JSONL history file that survives crashes.

License

MIT