tooleval

tooleval

npm audit for MCP servers — zero-config, zero-API-cost heuristic testing

ToolEval connects to any MCP server via stdio, discovers all tools, and runs 21 automated checks per tool covering schema validation, security, resilience, and correctness. No API keys. No LLM costs. Just plug in your server command and get results.

Quick Start

npx tooleval npx @modelcontextprotocol/server-filesystem /tmp

That's it. One command, full report.

Output Formats

Text (default)

npx tooleval npx @modelcontextprotocol/server-filesystem /tmp

🔍 ToolEval Spike — Generic MCP Server Checker
📦 Server: npx -y @modelcontextprotocol/server-memory

📋 Discovered 9 tools:
   • create_entities — Create multiple new entities in the knowledge graph
   • create_relations — Create multiple new relations between entities
   ...

────────────────────────────────────────────────────────────
🔧 Testing: create_entities
   ✅ A. Schema exists — 1 props
   ✅ B. Empty call (no crash) — error (expected)
   ✅ C. Response shape — 1 items
   ...

════════════════════════════════════════════════════════════
📊 SUMMARY
  ✅ create_entities: 18/18 checks
  Total: 162/162 checks passed
  ✅ ALL CLEAR
════════════════════════════════════════════════════════════

JSON

npx tooleval --format json npx @modelcontextprotocol/server-filesystem /tmp

Returns structured JSON with per-tool results — perfect for CI pipelines.

SARIF

npx tooleval --format sarif npx @modelcontextprotocol/server-filesystem /tmp

Outputs SARIF format — integrates with GitHub Code Scanning, Azure DevOps, and other SARIF-compatible tools.

HTML

npx tooleval --format html npx @modelcontextprotocol/server-filesystem /tmp > report.html

Self-contained HTML report — open in any browser, share with stakeholders.

The 21 Checks

#	Check	What it tests
A	Schema exists	Tool exposes a valid inputSchema object
B	Empty call resilience	Calling with {} doesn't crash the server
C	Response shape	Response has valid MCP content array
D	Response time	Responds within 10 seconds
E	Path traversal	Classic ../../../etc/passwd is rejected
F	Multi-vector traversal	6 path traversal bypass techniques blocked
G	Schema validation (Ajv)	inputSchema compiles as valid JSON Schema
H	Error info leakage	No stack traces, secrets, or paths in errors
I	Large input (1MB)	Server handles 1MB payloads gracefully
J	Concurrent resilience	5 simultaneous calls all return successfully
K	SSRF probe	Internal network URLs (169.254.x, localhost) are rejected
L	Command injection	Shell metacharacters in string params are rejected
M	Secret detection	Responses don't leak API keys, tokens, or credentials
N	Type coercion	Wrong types (string→int) are handled gracefully
O	Idempotency	Repeated identical calls produce consistent results
P	Timeout escalation	Slow inputs don't hang the server indefinitely
Q	Unicode handling	Unicode/special chars don't crash or corrupt
R	Required fields	Missing required fields produce proper errors
S	Nested depth	Deeply nested objects are handled gracefully
T	Enum boundary	Out-of-range enum values are rejected
U	Description quality	Tool has a meaningful description

Checks are automatically skipped when not applicable (e.g., path traversal skipped for tools without path params).

CI Integration

GitHub Actions

name: MCP Server Audit
on: [push, pull_request]

jobs:
  tooleval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - name: Run ToolEval
        run: npx tooleval --format json node ./dist/server.js > tooleval-report.json
      - name: Check results
        run: npx tooleval node ./dist/server.js
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: tooleval-report
          path: tooleval-report.json

GitHub Code Scanning (SARIF)

      - name: Run ToolEval (SARIF)
        run: npx tooleval --format sarif node ./dist/server.js > tooleval.sarif
      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: tooleval.sarif

Why ToolEval

• No API key needed — pure heuristic checks, no LLM calls
• No LLM cost — run it 1000 times in CI, it's free
• Tests the tool, not the AI — validates the MCP server interface directly
• Catches real security issues — path traversal, SSRF, command injection, info leakage
• Zero config — just point it at your server command
• CI-ready — JSON/SARIF/HTML output + proper exit codes

Exit Codes

Code	Meaning
0	All checks passed
1	Some checks failed
2	Fatal error (server failed to connect, etc.)

Requirements

• Node.js >= 18
• The MCP server must be launchable via a shell command (stdio transport)

License

MIT