@runhuman/mcp-server

Runhuman MCP Server

A Model Context Protocol (MCP) server that allows AI agents to interact with the Runhuman QA testing service.

Overview

This MCP server provides tools for creating and managing human QA jobs through the Runhuman API. AI agents can use this server to:

• Create new QA jobs with custom schemas
• Check the status of running jobs
• Retrieve completed job results

Installation

For Claude Desktop (Recommended)

• Get your API key at: https://runhuman.com/dashboard/api-keys

• Add to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on Mac):

{
  "mcpServers": {
    "runhuman": {
      "command": "npx",
      "args": ["-y", "@runhuman/mcp-server", "--api-key=qa_live_xxxxxxxxxxxxx"]
    }
  }
}

• Restart Claude Desktop

That's it! The server will be automatically downloaded and run by Claude.

For Development

From the monorepo root:

npm install
npm run build --workspace=@runhuman/mcp-server

# Run with API key
node packages/mcp-server/dist/index.js --api-key=qa_live_xxxxx

Available Tools

create_job

Create a new QA job with human testers.

Parameters:

• url (string): The URL to test
• description (string): Instructions for the human tester describing what to test
• schema (object): Expected result schema that the tester response will be extracted into
• targetDurationMinutes (number, optional): Time limit for tester (default: 5, range: 1-60)

wait_for_result

Check status, wait, and retrieve results for a QA job in a single convenient call.

Parameters:

• jobId (string): The ID of the job to check
• waitSeconds (number, optional): How long to wait before checking again (default: 30, range: 1-300)

Behavior:

• Checks status BEFORE waiting (returns immediately if already complete)
• Waits for the specified duration
• Checks status AFTER waiting
• Returns results if complete, otherwise suggests calling again with longer wait time

Usage Pattern:

// After creating a job, call repeatedly with increasing wait times:
let result = await wait_for_result(jobId, { waitSeconds: 30 });
if (result.status !== 'completed') {
  result = await wait_for_result(jobId, { waitSeconds: 45 });
}
if (result.status !== 'completed') {
  result = await wait_for_result(jobId, { waitSeconds: 60 });
}

Returns:

• result: Structured test results extracted from tester's response
• status: Job status (completed, failed, timeout, pending, claimed, in_progress)
• costUsd: Exact cost in USD with full precision (e.g., 0.396)
• testDurationSeconds: Time spent by tester in seconds (rounded up)
• testerResponse: Raw natural language feedback from the human tester
• testerAlias: Anonymized tester name (e.g., "Tester Alpha")
• testerAvatarUrl: Avatar image URL for UI display
• testerColor: Hex color code for theming (e.g., "#4A90E2")
• Additional metadata (timestamps, etc.)

Cost Calculation:

• Costs are calculated as: duration × $0.0018/second (general-use tier)
• Duration is always rounded UP using Math.ceil (any partial second counts)
• Costs are never $0 unless the tester never actually worked on it
• Full precision maintained (not rounded to cents)

Configuration

The MCP server needs to be configured with your Runhuman API credentials.

1. Get an API Key

Option A: Via Dashboard

• Start the API server: npm run dev --workspace=@runhuman/api
• Open http://localhost:3400/api-keys
• Click "Create API Key"
• Copy the key (starts with qa_live_)

Option B: Use Default Test Key

• For local development, you can use: qa_live_test_key_123
• This key exists in packages/api/data/api-keys.json

2. Configure Environment Variables

Create a .env file in the MCP server directory:

# For local development
RUNHUMAN_API_URL=http://localhost:3400
RUNHUMAN_API_KEY=qa_live_test_key_123

# For production
RUNHUMAN_API_URL=https://api.runhuman.com
RUNHUMAN_API_KEY=qa_live_xxxxxxxxxxxxxxxxxxxxx

Important: Never commit .env files to git! They're already in .gitignore.

3. Verify Configuration

Test your API key works:

curl http://localhost:3400/api/jobs \
  -H "Authorization: Bearer qa_live_test_key_123" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","description":"test","outputSchema":{}}'

Should return a job ID if authentication works.

For more details, see docs/API-AUTHENTICATION.md

Testing

The MCP server includes automated tests to verify it's working correctly:

# Build first
npm run build --workspace=@runhuman/mcp-server

# Run simple automated test
npm run test --workspace=@runhuman/mcp-server

# Or use the MCP Inspector (interactive testing)
npm run test:inspector --workspace=@runhuman/mcp-server

The test script will:

• ✅ Initialize a connection to the MCP server
• ✅ List all available tools (create_job, wait_for_result)
• ✅ Test calling the create_job tool

Expected Test Output

✅ Server initialized successfully
✅ Tools listed: create_job, wait_for_result
✅ create_job tool called successfully

Development

# Watch mode (auto-rebuild on changes)
npm run dev --workspace=@runhuman/mcp-server

# Build
npm run build --workspace=@runhuman/mcp-server

# Test after building
npm run test --workspace=@runhuman/mcp-server

Integration with Claude Desktop

To use this MCP server with Claude Desktop, add it to your configuration:

{
  "mcpServers": {
    "runhuman": {
      "command": "node",
      "args": ["/path/to/qa-experiment/packages/mcp-server/dist/index.js"]
    }
  }
}

Example Usage

Once connected to an AI agent (like Claude), the agent can use these tools naturally:

User: "Can someone test my checkout page at https://myapp.com/checkout?"

Agent uses create_job:

✅ Job created successfully!
Job ID: job_abc123
Status: pending
...

Agent calls wait_for_result repeatedly until complete:

⏳ Job Status: in_progress
Waited 30s, job not complete yet.
💡 Suggestion: Call wait_for_result again with waitSeconds: 45

Finally:

✅ Test completed!
Results Summary:
- Checkout Flow: ✅ Working
- Payment Processing: ✅ Successful
...

Developer Documentation

For developers working on this MCP server:

• docs/HOW-AGENTS-USE-MCP.md - How AI agents discover and use MCP servers
• docs/TOOL-RESPONSE-BEST-PRACTICES.md - Best practices for tool responses

Learn More

• Model Context Protocol Documentation
• Runhuman API Documentation