promptcheck

PromptCheck

PromptCheck is a CI-first test harness for LLM prompts. It lets you write tests for LLM outputs and automatically checks them in CI so you can catch prompt regressions early — before they reach users. PromptCheck works with any LLM provider, including OpenAI, Anthropic, and open-source models via Groq, OpenRouter, or local APIs.

PromptCheck is a CI-first test harness for LLM prompts.

Write tests in YAML, gate pull-requests, and see pass/fail summaries posted as comments—so your prompts don't quietly regress.

Install & Run

pip install promptcheck
promptcheck init   # creates a config and test scaffold
promptcheck run    # runs all prompt tests

Need full example? See example/ or the Quick-Start Guide.

Get Started

Ready to dive in?

• See a basic test example in the example/ directory.
• Follow our Quick-Start Guide for step-by-step setup instructions.

Why Prompt Testing Matters

LLMs can break without warning — even small prompt changes or model updates can cause major regressions. PromptCheck automates prompt evaluation like unit tests automate code quality.

What does PromptCheck do? Key Concepts in Automated LLM Evaluation

When you tweak a prompt, swap models, or refactor your agent code, PromptCheck runs a battery of tests in CI (Rouge, regex, token-cost, latency, etc.) and fails the pull-request if quality regresses or cost spikes.

Think pytest + coverage, but for LLM output.

Key Features for Effective LLM Testing

• Easy setup — drop a YAML test file, add the GitHub Action, done.
• Multi‑provider — Works with OpenAI, Anthropic, Groq, OpenRouter, or any model you connect via API (more built-in providers coming soon).
• Metrics out‑of‑the‑box — exact/regex match, ROUGE‑L, BLEU (optional), token‑count, latency, cost.
• Readable reports — Action log output and (coming soon) PR comment bot. run.json artifact produced.
• Fast to extend — write your own metric in <30 lines (standard Python).

Feature	Free	Pro
CLI & GitHub Action	✅	✅
Unlimited history & charts	—	✅
Slack alerts	—	✅

What it looks like

How the YAML works (tests/*.yaml)

A test file contains a list of test cases. Here's an example structure:

- id: "openrouter_greet_test_001"
  name: "OpenRouter Basic Greeting Test"
  description: "Tests a basic greeting prompt."
  type: "llm_generation"

  input_data:
    prompt: "Briefly introduce yourself and greet the user."

  expected_output:
    # For regex_match, a pattern to find in the LLM's output
    regex_pattern: ".+" # Example: matches any non-empty string

  metric_configs:
    - metric: "regex_match" 
    - metric: "token_count"
      parameters:
        count_types: ["completion", "total"]
    - metric: "latency"
      threshold: # Optional: fail if conditions aren't met
        value: 10000 # e.g., latency_ms <= 10000
    - metric: "cost" 

  model_config:
    provider: "openrouter"
    model_name: "mistralai/mistral-7b-instruct"
    parameters:
      temperature: 0.7
      max_tokens: 50
      timeout_s: 25.0 
      retry_attempts: 2

  tags: ["openrouter", "greeting"] 

Add more cases in tests/. Thresholds (like value for latency, or f_score for rouge) are defined within the threshold object of a metric_config.

Installation Options & Development Setup

# From PyPI (once 0.1.0+ is live)
# pip install promptcheck

# With optional BLEU metric (requires NLTK)
# pip install promptcheck[bleu]

# For development:
poetry install # Installs base dependencies
poetry install --extras bleu # Installs with BLEU support

Releasing (maintainers)

# 1. Ensure tests pass and docs are updated
# 2. Bump version in pyproject.toml
poetry version <new_version>  # e.g., 0.1.0, 0.2.0b1

# 3. Build the package
poetry build

# 4. Publish to TestPyPI first (configured in pyproject.toml or via poetry config)
# poetry config pypi-token.testpypi <YOUR_TESTPYPI_TOKEN>
poetry publish -r testpypi

# 5. Test TestPyPI package thoroughly (e.g., in a clean venv or CI)

# 6. Publish to PyPI (prod)
# poetry config pypi-token.pypi <YOUR_PYPI_TOKEN>
poetry publish

# 7. Tag the release in Git
git tag v<new_version>        # e.g., v0.1.0
git push origin v<new_version>

Documentation

📖 Docs: Quick-Start Guide · YAML Reference (Coming Soon!)

Roadmap

• PR comment bot (✅/❌ matrix in‑line)
• Hosted dashboard (Supabase)
• Async runner for large test suites
• More metrics and LLM provider integrations

Contributing

• Fork & clone the repository.
• Set up your development environment: poetry install --extras bleu (to include all deps).
• Run tests locally: poetry run promptcheck run tests/ (or a specific file). Keep it green!
• Make your changes, add tests for new features.
• Open a Pull Request.

Feedback & Questions

Found an issue or have a question? We'd love to hear from you! Please open an issue or start a discussion.

License

License: Business Source License 1.1
PromptCheck is free to use for evaluation and non-production use. For commercial licenses, contact us.

End of Document – keep this file as the project's living reference; version & timestamp changes at top on each major edit.