tracehub-mcp

tracehub-mcp

Also listed on the official MCP registry as io.github.mcpsmiths/tracehub-mcp (the registry's own ?search= endpoint can surface an outdated version first; this exact-name endpoint always reflects the current isLatest release).

Give your AI assistant a direct line into your observability backend. tracehub-mcp is an MCP (Model Context Protocol) server that lets Claude, Cursor, Windsurf, Gemini CLI, or any MCP client query OpenTelemetry traces from your LLM/GenAI application and reason about them — find expensive calls, debug errors, compare model performance, track token usage — without you copy-pasting trace JSON into a chat window.

It speaks OpenTelemetry's gen_ai.* semantic conventions natively, so it understands prompts, completions, token usage, and finish reasons as first-class concepts, not just generic span attributes.

tracehub-mcp started as a fork of traceloop/opentelemetry-mcp-server (Apache 2.0) — full attribution and fork history are in NOTICE. It's grown into a 5-backend, security-hardened server maintained independently under mcpsmiths; see What's Different From Upstream below for the parts that are new here.

Table of Contents

• Quick Start
• Supported Backends
• What's Different From Upstream
• Installation
• Configuration
• Diagnostics
• Security Considerations
• Privacy Policy
• MCP Client Setup
• Tools Reference
• Generic Filter System
• Example Queries
• Common Workflows
• Development
• Troubleshooting
• Roadmap
• License
• Support

Quick Start

tracehub-mcp is on PyPI. No install step needed — uvx fetches and runs it in one shot:

// claude_desktop_config.json
{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

Or from Claude Code directly:

claude mcp add tracehub-mcp -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://localhost:16686 -- uvx tracehub-mcp

That's it. Ask your assistant: "Show me traces with errors from the last hour."

See MCP Client Setup for Cursor, Windsurf, VS Code, and Gemini CLI, and Installation for pip/pipx/from-source alternatives.

Supported Backends

• Jaeger — local/self-hosted, the most common open-source trace backend. No auth required.
• Grafana Tempo — local or Grafana Cloud, TraceQL-native search.
• Traceloop — cloud LLM observability platform, API-key auth.
• Datadog — cloud APM, requires an API key and an Application key.
• Sentry — cloud or self-hosted, requires an auth token and an organization slug.
• AWS X-Ray — SigV4-signed via boto3, requires an AWS region and standard AWS credentials (not an API key). Because OTel span attributes land in unindexed X-Ray segment metadata by default (only annotations are queryable), native server-side filtering is narrower here than for the other backends — see the Backend Support Matrix.
• New Relic — NerdGraph GraphQL API, requires a User API key (not an Ingest/License key) and an account ID. Built without a live account to verify against — see the New Relic setup section below and the module docstring in backends/newrelic.py for exactly which assumptions are unverified.
• Honeycomb — Query Data API, requires a Configuration Key and a dataset slug. Built without a live account to verify against, and the research behind it found the programmatic Query Data API (the only way to actually run a query and get results back) is documented as Enterprise-plan exclusive — see the Honeycomb setup section below and the module docstring in backends/honeycomb.py.

All eight implement the same BaseBackend interface, so every MCP tool works identically regardless of which one you point the server at. See Configuration for per-backend setup.

What's Different From Upstream

Upstream opentelemetry-mcp-server shipped Jaeger, Tempo, and Traceloop. tracehub-mcp adds Datadog and Sentry as full backends — not thin wrappers, but complete implementations of every tool (search, span search, trace hydration, service discovery, health checks). Along the way, all five backends — including the three inherited from upstream — were hardened to a consistent bar:

• HTTPS-only enforcement on cloud backends. Datadog and Sentry both refuse to start against a plain http:// URL, because their auth is a bearer token / API+App key pair that has no business going out over plaintext.
• Query-injection-safe escaping. Every value spliced into a Datadog span-search query or a Sentry Discover query is escaped and exact-quoted; field names (which are less obviously untrusted, since they come from the MCP tool's filters parameter) are validated against an allowlist pattern before being spliced into the query string, closing off structural injection through a crafted field name.
• Bounded pagination on every backend that paginates via cursor (Datadog, Sentry) — a search stops at the requested limit or when the backend stops returning a continuation cursor, whichever comes first, so a single tool call can't degrade into an unbounded crawl.
• Exact-ID re-verification. Where a backend's search API can return neighbors instead of an exact match (notably Datadog's trace reconstruction from grouped spans), every result is re-checked against the exact ID that was asked for before being returned.
• No fabricated data on malformed responses, in the backends we built. Datadog and Sentry reject a span outright — rather than substituting a placeholder like now() for a missing timestamp or a literal "unknown" for a missing service_name/operation_name — since a fabricated value would silently corrupt trace ordering, duration aggregation, and any tool that groups by service or operation. (The three backends inherited from upstream — Jaeger, Tempo, Traceloop — predate this discipline and haven't been retrofitted; that's deliberate scope discipline, not an oversight, mirroring this project's own precedent of not reaching into shared/inherited code without full regression coverage for it.)

All of this is backed by 458 passing tests (2 skipped, zero regressions), a clean ruff check and mypy --strict run, and two rounds of adversarial CodeRabbit review on the new backends.

Installation

tracehub-mcp is on PyPI. Pick whichever of these your workflow already uses — they're equivalent.

Option 1: uvx (no install step)

uvx tracehub-mcp --backend jaeger --url http://localhost:16686

This is what the Quick Start config above uses — uv fetches the package and runs the tracehub-mcp entry point in one shot, nothing left behind on disk between runs.

Option 2: pip / pipx

pipx install tracehub-mcp
# or: pip install tracehub-mcp

tracehub-mcp --backend jaeger --url http://localhost:16686

Option 3: Clone and run from source

git clone https://github.com/mcpsmiths/tracehub-mcp.git
cd tracehub-mcp
uv sync

uv run tracehub-mcp --backend jaeger --url http://localhost:16686

Use this if you're developing locally, want to pin to a specific commit, or want the dev tooling installed (uv sync --group dev).

Option 4: Docker

docker run --rm -p 8000:8000 \
  -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://host.docker.internal:16686 \
  ghcr.io/mcpsmiths/tracehub-mcp:latest

Runs HTTP transport by default (the image's CMD); clients connect to http://localhost:8000/mcp. This is also the form to use for MCP clients whose config takes a command/args pair pointing at docker directly (Cursor, Windsurf) instead of a local binary.

Prerequisites: Python 3.11+, plus uv for Options 1 and 3; Docker for Option 4.

Configuration

Configuration comes from environment variables, CLI flags, or both. Precedence: CLI arguments > environment variables > defaults.

# .env (see .env.example)
BACKEND_TYPE=jaeger
BACKEND_URL=http://localhost:16686

# Equivalent via CLI flags
tracehub-mcp --backend jaeger --url http://localhost:16686

All Configuration Options

Variable	Type	Default	Description
BACKEND_TYPE	string	jaeger	Backend type: jaeger, tempo, traceloop, datadog, sentry, xray, newrelic, or honeycomb
BACKEND_URL	URL	-	Backend API endpoint (required; decorative-only placeholder for X-Ray)
BACKEND_API_KEY	string	-	API key/auth token (required for Traceloop, Datadog, Sentry, New Relic, and Honeycomb; unused by X-Ray)
BACKEND_APP_KEY	string	-	Application key (Datadog only, in addition to BACKEND_API_KEY)
BACKEND_TEMPO_INSTANCE_ID	string	-	Grafana Cloud stack/instance ID (Tempo only, enables Basic Auth in addition to BACKEND_API_KEY)
BACKEND_SENTRY_ORG	string	-	Organization slug (required for Sentry)
BACKEND_SENTRY_PROJECT	string	-	Project slug (optional for Sentry, narrows queries to one project)
BACKEND_AWS_REGION	string	-	AWS region, e.g. us-east-1 (required for X-Ray)
BACKEND_NEWRELIC_ACCOUNT_ID	string	-	New Relic account ID (required for New Relic - NerdGraph queries are user-scoped, not account-scoped)
BACKEND_HONEYCOMB_DATASET	string	-	Honeycomb dataset slug (required for Honeycomb - the Query Data API is dataset-scoped)
BACKEND_ENVIRONMENTS	string	prd	Comma-separated environments (Traceloop only)
BACKEND_TIMEOUT	float	30	Request timeout in seconds
SECONDARY_BACKEND_TYPE / SECONDARY_BACKEND_URL / ...	string/URL	unset (opt-in)	A second, independently-configured backend for correlate_trace (cross-backend correlation) - every BACKEND_* variable above has a SECONDARY_BACKEND_* equivalent. Env-var only, no CLI flags. Unset means correlate_trace raises a clear error if called
LOG_LEVEL	string	INFO	Logging level: DEBUG, INFO, WARNING, ERROR (--log-level)
MAX_TRACES_PER_QUERY	integer	500	Server-wide ceiling (1-1000, --max-traces-per-query) - caps every tool's limit argument before it reaches a backend query, regardless of what the calling agent requests
SLOW_REQUEST_THRESHOLD_MS	float	unset	Logs a WARNING for any backend request slower than this, independent of LOG_LEVEL (--slow-request-threshold-ms)
MCP_TRANSPORT / MCP_HOST / MCP_PORT	string/int	stdio/0.0.0.0/8000	Env-var equivalents of --transport/--host/--port
MCP_INCLUDE_ARGS_IN_SPANS	bool	false	Include tool call arguments/results as OTel span attributes when self-instrumentation is enabled below - off by default since they may contain sensitive data. Known credential shapes (Bearer tokens, api_key=/secret=/password=-style fields, AWS/GitHub/common vendor key prefixes) are redacted before export, but this is a pattern match, not a guarantee - trace_id/span_id and other legitimate trace data are deliberately left untouched (--include-args-in-spans)
OTEL_EXPORTER_OTLP_ENDPOINT	URL	unset	Enables opt-in OTel self-instrumentation of tool calls when set; unset means zero overhead (no TracerProvider configured, no middleware registered)
OTEL_SERVICE_NAME	string	tracehub-mcp	Service name reported in self-instrumentation spans
QUERY_CACHE_TTL_SECONDS	float	unset	Cache backend query results (search_traces/search_spans/get_trace/list_services/get_service_operations) for this many seconds, with in-flight request coalescing (unset: disabled, --query-cache-ttl-seconds)
RATE_LIMIT_MAX_REQUESTS	integer	100	HTTP transport only. Max requests per client IP per RATE_LIMIT_WINDOW_SECONDS (--rate-limit-max-requests, set to 0 to disable)
RATE_LIMIT_WINDOW_SECONDS	float	60	HTTP transport only. Fixed window size in seconds for RATE_LIMIT_MAX_REQUESTS (--rate-limit-window-seconds)
SHUTDOWN_DRAIN_SECONDS	float	0.0	HTTP transport only. On shutdown, wait this many seconds inside the ASGI shutdown handler - after uvicorn has already finished draining in-flight connections - before closing the shared backend HTTP client (--shutdown-drain-seconds)
BACKEND_CLOSE_TIMEOUT_SECONDS	float	5.0	HTTP transport only. Abandon closing the shared backend HTTP client during shutdown if it does not finish within this many seconds - uvicorn places no timeout of its own around this wait (--backend-close-timeout-seconds)
GRACEFUL_SHUTDOWN_TIMEOUT_SECONDS	integer	2	HTTP transport only. uvicorn's own bound (whole seconds only) on waiting for in-flight connections/tasks to finish on shutdown before cancelling them (--graceful-shutdown-timeout-seconds)

Every backend-related CLI flag has a matching env var (--backend/BACKEND_TYPE, --url/BACKEND_URL, --api-key/BACKEND_API_KEY, --app-key/BACKEND_APP_KEY, --tempo-instance-id/BACKEND_TEMPO_INSTANCE_ID, --sentry-org/BACKEND_SENTRY_ORG, --sentry-project/BACKEND_SENTRY_PROJECT, --aws-region/BACKEND_AWS_REGION, --environments/BACKEND_ENVIRONMENTS). --disable-tools <name1,name2,...> / --enabled-tools <name1,name2,...> (CLI-only, no env var) remove/allowlist tools for reduced-trust deployments - --enabled-tools is applied first, --disable-tools on top of whatever it kept. Run tracehub-mcp --help for the full list.

Known third-party egress dependency: the underlying FastMCP framework checks PyPI (https://pypi.org/pypi/fastmcp/json) for a newer FastMCP release once every 12 hours when it prints its startup banner - this is FastMCP's own behavior, not tracehub-mcp's, and unrelated to the OTel self-instrumentation above. It fails silently if there's no network access. For network-restricted/air-gapped deployments, disable it with FASTMCP_CHECK_FOR_UPDATES=off, or suppress the banner entirely with FASTMCP_SHOW_SERVER_BANNER=false.

Backend-Specific Setup

Jaeger

BACKEND_TYPE=jaeger
BACKEND_URL=http://localhost:16686

No API key required. search_traces and search_spans_tool both require a service_name parameter — Jaeger's API is optimized for per-service queries, so querying across all services isn't supported. Discover service names first with list_services.

Grafana Tempo

BACKEND_TYPE=tempo
BACKEND_URL=http://localhost:3200

No API key required for a local/self-hosted install. Search uses TraceQL under the hood; service_name is optional.

For Grafana Cloud-hosted Tempo, also set BACKEND_TEMPO_INSTANCE_ID to the stack's instance ID and BACKEND_API_KEY to a Cloud Access Policy token scoped to traces:read — Grafana Cloud requires Basic Auth (instance ID as username, token as password) instead of self-hosted Tempo's Bearer-token auth:

BACKEND_TYPE=tempo
BACKEND_URL=https://tempo-prod-XX-prod-XX-XXXX.grafana.net
BACKEND_TEMPO_INSTANCE_ID=your_stack_instance_id
BACKEND_API_KEY=your_cloud_access_policy_token

Traceloop

BACKEND_TYPE=traceloop
BACKEND_URL=https://api.traceloop.com
BACKEND_API_KEY=your_api_key_here

The API key encodes project information — the backend always uses a project slug of "default", and Traceloop resolves the actual project/environment from the key itself.

Datadog

BACKEND_TYPE=datadog
# US site (default): https://api.datadoghq.com
# EU site:            https://api.datadoghq.eu
BACKEND_URL=https://api.datadoghq.com
BACKEND_API_KEY=your_api_key_here
BACKEND_APP_KEY=your_application_key_here

Datadog requires both an API key and an Application key — a single key is not enough for span/trace queries, even though ingestion only needs the API key. Trace search uses Datadog's span search query syntax rather than TraceQL or Jaeger-style tag params, and traces are reconstructed from grouped spans since Datadog has no trace-level lookup endpoint. The backend also refuses a plain http:// URL — see What's Different From Upstream.

Troubleshooting: a 403 from the Datadog API almost always means the Application key (not the API key) is missing or invalid. If you're on the EU site, double check BACKEND_URL is https://api.datadoghq.eu, not the US default.

Sentry

BACKEND_TYPE=sentry
# SaaS (may be region-specific, e.g. https://us.sentry.io):
BACKEND_URL=https://sentry.io
BACKEND_API_KEY=your_auth_token_here
BACKEND_SENTRY_ORG=your-org-slug
# Optional: narrow queries to one project
BACKEND_SENTRY_PROJECT=your-project-slug

Sentry requires both an auth token and an organization slug — every endpoint this backend calls is organization-scoped. Trace search uses Sentry's search syntax against the Discover/Explore Events API. Unlike Datadog, Sentry does have a native trace-lookup endpoint, so get_trace calls it directly instead of reconstructing a trace from spans — search_traces still discovers candidate trace IDs via a span search first, since Sentry's search surface is itself span-centric. Like Datadog, this backend refuses a plain http:// URL.

Troubleshooting: a 403/401 from the Sentry API almost always means the auth token is missing, invalid, or lacks the necessary scopes. A 404 on an org-scoped endpoint usually means the organization slug is wrong. For a self-hosted install, BACKEND_URL should be the install's own base URL, not https://sentry.io. Some of the tracing endpoints this backend depends on are newer/experimental on Sentry's side and may not be available on every plan or self-hosted version — see the module docstring in backends/sentry.py for specifics.

AWS X-Ray

BACKEND_TYPE=xray
# Decorative only - never dereferenced for a live request. Keep it
# consistent with BACKEND_AWS_REGION by convention.
BACKEND_URL=https://xray.us-east-1.amazonaws.com
BACKEND_AWS_REGION=us-east-1

Unlike every other backend here, X-Ray is queried via boto3/SigV4, not a bearer token — auth comes from boto3's standard credential chain (environment variables, ~/.aws/credentials, an assumed role, or an instance/task role). The running process needs xray:GetTraceSummaries, xray:BatchGetTraces, and (optionally, for faster list_services) xray:GetServiceGraph IAM permissions.

Filtering is more limited than the other backends. By default, OpenTelemetry span attributes are converted to X-Ray segment metadata, not annotations — only annotations are indexed and queryable via X-Ray's FilterExpression search syntax. This backend can only natively push down service name, duration, and error/fault/throttle-derived status; every other field (including all gen_ai.* attributes) is always applied client-side after hydrating full traces, unless your own OTel/ADOT collector config explicitly promotes those keys to indexed annotations.

Troubleshooting: an AccessDeniedException from GetServiceGraph is non-fatal — list_services automatically falls back to sampling recent traces. An AccessDeniedException from GetTraceSummaries/BatchGetTraces is fatal for search/hydration and will surface as an unhealthy health_check/doctor result; double check the IAM permissions above.

New Relic

BACKEND_TYPE=newrelic
# US datacenter (default): https://api.newrelic.com/graphql
# EU datacenter:            https://api.eu.newrelic.com/graphql
BACKEND_URL=https://api.newrelic.com/graphql
BACKEND_API_KEY=your_user_api_key_here
BACKEND_NEWRELIC_ACCOUNT_ID=1234567

New Relic requires a User API key (not an Ingest or License key, which cannot query) and an account ID — a User key is user-scoped, not account-scoped, so every NerdGraph query here states which account to query explicitly. Search uses NRQL (SELECT * FROM Span WHERE ...); get_trace uses NerdGraph's native distributedTracing.trace lookup, like Sentry, rather than reconstructing a trace from search results.

Built without a live New Relic account to verify against. Two things a real account is needed to confirm: the exact shape of OTel span attributes inside distributedTracing.trace's GraphQL response (assumed to be a flat, directly-keyed object), and the account's actual Span-data retention window (a conservative 7-day default is used pending confirmation). See the module docstring in backends/newrelic.py for the complete list of unverified assumptions before relying on this backend in production.

Honeycomb

BACKEND_TYPE=honeycomb
# US instance (default): https://api.honeycomb.io
# EU instance:            https://api.eu1.honeycomb.io
BACKEND_URL=https://api.honeycomb.io
BACKEND_API_KEY=your_configuration_key_here
BACKEND_HONEYCOMB_DATASET=your-dataset-slug

Honeycomb requires a Configuration Key (not an Ingest or Management key) with "Manage Queries and Columns" and "Run Queries" permissions, and a dataset slug — the Query Data API is dataset-scoped. Search runs Honeycomb's 3-step async query flow (create a query spec, run it, poll for the result) rather than a text query language like NRQL/TraceQL; get_trace searches by trace.trace_id since no dedicated single-trace lookup endpoint is documented.

Go/no-go risk, not just an unverified detail. The research behind this backend found the programmatic Query Data API — create-query-result + get-query-result, the only way to actually run a query and get data back — is documented as Enterprise-plan exclusive; Free and Pro accounts can build a query specification via the API but never execute it. This backend raises a clear HoneycombEnterpriseRequiredError if a 402/403 is returned, but the exact status code a non-Enterprise account gets back was never confirmed live, and this backend was built entirely without a live account (Enterprise or otherwise). See the module docstring in backends/honeycomb.py for the complete list of unverified assumptions - including how raw per-span rows are extracted from Honeycomb's aggregation-oriented Query API - before relying on this backend in production.

Transport Modes

# stdio (default) — local use, Claude Desktop, single process
uvx tracehub-mcp
tracehub-mcp                      # pipx/pip install
uv run tracehub-mcp               # from-source install

# HTTP — remote access, multiple clients, network deployment, sample applications
uvx tracehub-mcp --transport http --host 0.0.0.0 --port 8000
tracehub-mcp --transport http --host 0.0.0.0 --port 8000              # pipx/pip install
uv run tracehub-mcp --transport http --host 0.0.0.0 --port 8000       # from-source install

With HTTP transport, clients connect to http://<host>:<port>/mcp (streamable-HTTP, for compatibility across MCP clients).

Fly.io: the one-click fly mcp launch command only supports stdio-transport servers — it can't deploy this server's --transport http mode. Running tracehub-mcp on Fly.io with HTTP transport needs the manual fly.toml + fly deploy path instead.

Diagnostics

Validate a config before wiring it into a real MCP client:

tracehub-mcp doctor --backend jaeger --url http://localhost:16686

[OK] Configuration loaded and validated
[OK] Backend constructed: jaeger @ http://localhost:16686/
[OK] Health check: healthy
[OK] Connectivity probe (list_services): 2 service(s)

doctor accepts the same --backend/--url/etc. flags as the main command and exits non-zero if any step fails - unlike normal server startup, which lazily initializes the backend and deliberately keeps running even if the initial health check fails.

To see the fully-resolved configuration (env vars + CLI overrides merged) without starting the server:

tracehub-mcp --print-config --backend jaeger --url http://localhost:16686

Secrets (--api-key/--app-key) are reported as api_key_set/app_key_set booleans, never their actual value.

Security Considerations

Trace and span data returned by this server — attribute values, error messages, operation names — comes from whatever application your observability backend is instrumenting, not from tracehub-mcp itself. That makes it fundamentally the same category of untrusted external content as a webpage or a file, even though it's a trusted server (this one) handing it back to your MCP client.

• Treat backend data as untrusted input. An LLM client consuming trace/span data from tracehub-mcp should apply the same caution it would to any other external tool output — a span attribute or error message is application data to reason about, not an instruction to follow, no matter how it's phrased.
• This is a known MCP risk category, not a tracehub-mcp-specific one. OWASP's GenAI Security Project covers it in their Practical Guide for Secure MCP Server Development, and Anthropic's own engineering guidance, How We Contain Claude, states plainly that tool output is an attack surface even when the tool itself is trusted.
• Practical implication: if you're querying traces from an application that processes untrusted user input (e.g. a customer-facing chatbot), be aware that adversarial content a user fed into that application could end up in a span attribute this server returns — and from there, in your LLM client's context.

Privacy Policy

tracehub-mcp is self-hosted software with no maintainer-operated service or account — it collects no data of its own. See PRIVACY.md for the full policy: what data the software touches, where network calls go, and how secrets and self-instrumentation are handled.

MCP Client Setup

Every example below uses uvx tracehub-mcp (no install step). Swap in tracehub-mcp (pip/pipx install) or uv run tracehub-mcp (from-source, --directory /absolute/path/to/tracehub-mcp) if you installed it a different way — see Installation.

Claude Desktop

Config file location:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

Jaeger (no auth):

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

Datadog (API key + App key):

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "datadog",
        "BACKEND_URL": "https://api.datadoghq.com",
        "BACKEND_API_KEY": "your_api_key_here",
        "BACKEND_APP_KEY": "your_application_key_here"
      }
    }
  }
}

Sentry (auth token + org slug):

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "sentry",
        "BACKEND_URL": "https://sentry.io",
        "BACKEND_API_KEY": "your_auth_token_here",
        "BACKEND_SENTRY_ORG": "your-org-slug"
      }
    }
  }
}

If you're running from a clone instead, the bundled wrapper script gives easy backend switching during local dev:

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "/absolute/path/to/tracehub-mcp/start_locally.sh"
    }
  }
}

(the script ships Jaeger/Traceloop/Tempo blocks only, with Jaeger active by default — to switch, comment out the active block and uncomment the one you want; Datadog/Sentry aren't in the script, so add their export lines manually).

Claude Code

claude mcp add tracehub-mcp -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://localhost:16686 -- uvx tracehub-mcp

Datadog/Sentry work the same way — add more -e KEY=value flags for each backend's required env vars (see Backend-Specific Setup). Then:

claude mcp list
claude "Show me traces with errors from the last hour"

Cursor

.cursor/mcp.json (project-level) or your global Cursor MCP config:

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

Windsurf

~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

VS Code (GitHub Copilot)

.vscode/mcp.json in your workspace — note the top-level key is servers, not mcpServers, and stdio servers need no "type" field:

{
  "servers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

Gemini CLI

Config file: ~/.gemini/config.json, same JSON shape as Claude Desktop above. Then:

gemini "Analyze token usage for gpt-4 requests today"

Tools Reference

tracehub-mcp exposes 19 MCP tools:

Tool	Description	Use Case
search_traces	Search traces with simple params or advanced filters	Find specific requests or patterns
search_spans_tool	Search individual spans (not grouped into traces)	Find LLM tool calls, specific ops
get_trace	Get complete trace details by trace ID	Deep-dive into a single trace
triage_trace	Synthesize a likely-root-cause diagnosis for a trace (critical path, latency ranking, error chain)	Act on a diagnosis instead of re-deriving one from a trace dump
correlate_trace	Find the corresponding trace in a second, independently-configured backend	Join a trace across two backends (e.g. Datadog → Sentry)
get_llm_usage	Aggregate token usage metrics	Track costs and usage trends
list_services	List available services	Discover what's instrumented
find_errors	Find traces with errors	Debug failures quickly
list_llm_models	Discover models in use, with usage stats	Track model adoption, shadow AI
get_llm_model_stats	Latency/token percentiles + finish reasons for one model	Compare model efficiency
get_llm_expensive_traces	Find highest token-usage traces	Cost optimization
get_llm_slow_traces	Find slowest traces by duration	Latency debugging
list_llm_tools_tool	Discover LLM tool/function calls (traceloop.span.kind == tool)	Track agent tool usage
list_sessions	Group spans by gen_ai.conversation.id	Understand multi-turn conversation activity
get_session_stats	Detailed stats for one conversation ID	Drill into a single conversation
compare_time_windows	Diff aggregated usage between two time ranges	"This week vs last week" comparisons
investigate_cost_spike	Compare cost between a recent window and a baseline, ranked by model/service	"Why did our LLM bill spike?"
investigate_error_spike	Compare error rate between a recent window and a baseline, ranked by service/model/error type	"Is this error increase a real spike?"
get_prompt_version_stats	Group spans by gen_ai.prompt.name/.version	Compare prompt versions before promoting one

Backend Support Matrix

Feature	Jaeger	Tempo	Traceloop	Datadog	Sentry	AWS X-Ray	New Relic	Honeycomb♦
Search traces	✓	✓	✓	✓†	✓‡	✓†	✓†	✓†
Search spans	✓*	✓	✓	✓	✓	✓†	✓	✓
Get trace by ID	✓	✓	✓	✓†	✓	✓	✓‡	✓†
Advanced filters	✓	✓	✓	✓	✓	✓§	✓	✓
Error traces	✓	✓	✓	✓	✓	✓	✓	✓
All LLM tools	✓	✓	✓	✓	✓	✓	✓¶	✓¶

_{* Jaeger requires the service_name parameter for span search.
† Datadog, AWS X-Ray, New Relic, and Honeycomb have no trace-level search API (X-Ray's BatchGetTraces fetches by exact ID); traces are reconstructed by searching (spans, X-Ray trace summaries, NRQL Span events, or Honeycomb breakdown groups) and hydrating, with every result re-verified against the exact ID requested.
‡ Sentry and New Relic both have a native trace-lookup endpoint (unlike Datadog and Honeycomb), so get_trace calls it directly; search_traces still discovers candidate trace IDs via a span/NRQL search first, since both backends' search surface is itself span-centric.
§ AWS X-Ray natively filters only service name, duration, and error/fault/throttle-derived status — OTel attributes (including all gen_ai.* fields) land in unindexed segment metadata by default, not indexed annotations, so they're always applied client-side rather than pushed down to FilterExpression.
¶ New Relic: search_spans reads gen_ai.* attributes directly off NRQL's well-documented flat row shape; search_traces/get_trace additionally rely on an unverified assumption about the shape of the attributes field inside NerdGraph's distributedTracing.trace response, mitigated by overlaying the original search row's attributes back onto the hydrated result. Honeycomb: gen_ai.* attributes are extracted via the same breakdown-column technique used for every other field (module docstring point 2), an equally unverified extraction method.
♦ Honeycomb's entire "Search"/"Get trace by ID" row of checkmarks above depends on the Query Data API, which the research behind this backend found is documented as Enterprise-plan exclusive - a Free/Pro account may not be able to exercise any of this at all. See the Honeycomb setup section above.}

Key Tool Details

search_traces

{
  "service_name": "my-app",
  "start_time": "2024-01-01T00:00:00Z",
  "end_time": "2024-01-01T23:59:59Z",
  "gen_ai_system": "openai",
  "gen_ai_request_model": "gpt-4",
  "min_duration_ms": 1000,
  "has_error": false,
  "limit": 50
}

Parameters: service_name, operation_name, start_time/end_time (ISO 8601), min_duration_ms/max_duration_ms, gen_ai_system, gen_ai_request_model, gen_ai_response_model, has_error, tags, filters (see Generic Filter System), limit (1-1000, default 100). Returns trace summaries with token counts.

get_trace

{ "trace_id": "abc123def456" }

Returns the full trace tree: all spans with attributes, parsed OpenTelemetry gen_ai.* data for LLM spans, per-span token usage, error information, and each span's raw events (e.g. gen_ai.evaluation.result, or any other instrumentation-emitted event — not filtered to a fixed set of names).

triage_trace

{ "trace_id": "abc123def456", "detail_level": "summary" }

Instead of returning raw trace data for an agent to re-derive a diagnosis from every time, this synthesizes one directly: a critical path (the "Last Finishing Child" chain actually responsible for the trace's total latency), the top spans ranked by self-time (latency contribution net of children), and — when the trace contains an error — the deepest error span in the trace's error chain as the likely root cause (confidence: "high", or "medium" when multiple equally-deep error chains make blame ambiguous). Falls back to the highest self-time span as a pure-latency diagnosis (confidence: "low") when no error is present. Deterministic — no LLM call — and works against any configured backend, since it operates on get_trace's already-fetched span data. detail_level: "full" additionally attaches the diagnosed root cause's raw error detail (message/type/stacktrace) when the verdict is error-driven.

correlate_trace

{ "trace_id": "abc123def456" }

Tries to find the corresponding trace in a second, independently-configured backend (e.g. a Datadog trace and its downstream Sentry error, joined). Tries a direct trace_id match in the secondary backend first (confidence: "high"); if that fails, falls back to a time-window + service-name-overlap heuristic search (confidence: "low") - queried once per service name in the primary trace, so this works against backends like Jaeger that require service_name on search_traces. Each match also reports root_cause_consistent: null when neither trace has an error to compare, otherwise true/false for whether both sides agree on the error and which service it traces back to (reusing the same error-chain logic triage_trace uses). This is a best-effort correlation, not a guaranteed join - the result always includes a fixed limitations list (clock skew, sampling mismatches, partial trace visibility, and the fact that trace-id continuity across a vendor boundary isn't guaranteed even under normal W3C Trace Context propagation). Requires a secondary backend configured via SECONDARY_BACKEND_TYPE/SECONDARY_BACKEND_URL (see configuration table); raises a clear error otherwise.

get_llm_usage

{
  "start_time": "2024-01-01T00:00:00Z",
  "end_time": "2024-01-01T23:59:59Z",
  "service_name": "my-app",
  "gen_ai_system": "openai",
  "limit": 1000
}

Returns aggregated prompt/completion/total tokens, broken down by model and by service, plus request counts.

list_services — no parameters. Returns the list of instrumented service names.

find_errors

{
  "start_time": "2024-01-15T14:00:00Z",
  "service_name": "my-app",
  "limit": 50
}

Returns error messages, error types, truncated stack traces, and LLM-specific error info.

list_llm_models / get_llm_model_stats / get_llm_expensive_traces / get_llm_slow_traces / list_llm_tools_tool / search_spans_tool / list_sessions / get_session_stats / compare_time_windows / investigate_cost_spike / investigate_error_spike / get_prompt_version_stats are documented in detail, with worked examples, in CLAUDE.md — this README covers the shape every tool shares; CLAUDE.md is the fuller reference for exact parameters and response fields on the LLM-analysis tools.

Generic Filter System

search_traces and search_spans_tool both accept a filters list in addition to (or instead of) their simple named parameters, for advanced queries. Each filter is:

{
  "field": "gen_ai.usage.total_tokens",
  "operator": "gt",
  "value": 5000,
  "value_type": "number"
}

• field — dotted attribute name, e.g. gen_ai.usage.prompt_tokens, traceloop.span.kind, service.name
• operator — see table below
• value — single value (most operators) or values — list (for in, not_in, between)
• value_type — "string", "number", or "boolean"

Category	Operators
String	equals, not_equals, contains, not_contains, starts_with, ends_with, in, not_in
Number	equals, not_equals, gt, lt, gte, lte, between, in, not_in
Boolean	equals, not_equals
Existence	exists, not_exists (no value needed)

Multiple filters combine with AND logic. Legacy simple parameters (service_name, gen_ai_request_model, etc.) still work and are converted to filters internally — mix and match freely.

The server uses a hybrid filtering strategy: filters are pushed to the backend's native query language when supported (TraceQL for Tempo, span-search syntax for Datadog, Discover syntax for Sentry), and applied client-side afterward for anything the backend can't express natively.

Backend	Native filter support	Notes
Tempo (TraceQL)	equals, not_equals, gt, lt, gte, lte, contains (regex), in (OR), exists, not_exists	—
Traceloop	equals, not_equals, gt, lt, gte, lte	—
Datadog	Most operators via span-search syntax	Field names validated against an allowlist before being spliced into the query
Sentry	Most operators via Discover search syntax	Same field-name allowlisting as Datadog
Jaeger	equals (via tags only)	Requires service_name

Example — expensive OpenAI traces:

{
  "filters": [
    { "field": "gen_ai.system", "operator": "equals", "value": "openai", "value_type": "string" },
    { "field": "gen_ai.usage.total_tokens", "operator": "gt", "value": 5000, "value_type": "number" }
  ]
}

For the full semantic-convention attribute list (gen_ai.* vs legacy llm.*, token-naming variants across providers, finish-reason values, and the token-calculation fallback chain), see CLAUDE.md.

Example Queries

Find Expensive OpenAI Operations

Ask: "Show me OpenAI traces from the last hour that took longer than 5 seconds"

Tool call: search_traces

{
  "service_name": "my-app",
  "gen_ai_system": "openai",
  "min_duration_ms": 5000,
  "start_time": "2024-01-15T10:00:00Z",
  "limit": 20
}

Response:

{
  "traces": [
    {
      "trace_id": "abc123...",
      "service_name": "my-app",
      "operation_name": "chat.completions",
      "status": "OK",
      "duration_ms": 8250,
      "span_count": 3,
      "llm_span_count": 1,
      "total_tokens": 4523,
      "has_errors": false
    }
  ],
  "count": 1
}

Analyze Token Usage by Model

Ask: "How many tokens did we use for each model today?"

Tool call: get_llm_usage

{
  "start_time": "2024-01-15T00:00:00Z",
  "end_time": "2024-01-15T23:59:59Z",
  "service_name": "my-app"
}

Response:

{
  "period": { "start_time": "2024-01-15T00:00:00Z", "end_time": "2024-01-15T23:59:59Z" },
  "filters": { "service_name": "my-app" },
  "summary": {
    "total_requests": 487,
    "total_prompt_tokens": 82140,
    "total_completion_tokens": 43290,
    "total_tokens": 125430
  },
  "by_model": {
    "gpt-4": { "requests": 156, "prompt_tokens": 58300, "completion_tokens": 26900, "total_tokens": 85200 },
    "gpt-3.5-turbo": { "requests": 331, "prompt_tokens": 23840, "completion_tokens": 16390, "total_tokens": 40230 }
  },
  "by_service": {
    "my-app": { "requests": 487, "prompt_tokens": 82140, "completion_tokens": 43290, "total_tokens": 125430 }
  }
}

Find Traces with Errors

Ask: "Show me all errors from the last hour"

Tool call: find_errors

{
  "start_time": "2024-01-15T14:00:00Z",
  "service_name": "my-app",
  "limit": 10
}

Response:

{
  "count": 1,
  "error_traces": [
    {
      "trace_id": "def456...",
      "service_name": "my-app",
      "operation_name": "chat.completions",
      "start_time": "2024-01-15T14:23:15Z",
      "duration_ms": 1200,
      "status": "ERROR",
      "span_count": 2,
      "llm_span_count": 1,
      "total_tokens": 310,
      "has_errors": true,
      "error_spans": [
        {
          "span_id": "span789...",
          "operation_name": "chat.completions",
          "service_name": "my-app",
          "status": "ERROR",
          "error_message": "RateLimitError: Too many requests",
          "error_type": "openai.error.RateLimitError",
          "is_llm_error": true,
          "llm_provider": "openai",
          "llm_model": "gpt-4"
        }
      ]
    }
  ]
}

Compare Model Performance

Ask: "What's the performance difference between GPT-4 and Claude?"

Tool call 1: get_llm_model_stats for gpt-4

{ "model_name": "gpt-4", "start_time": "2024-01-15T00:00:00Z" }

Tool call 2: get_llm_model_stats for claude-3-opus

{ "model_name": "claude-3-opus-20240229", "start_time": "2024-01-15T00:00:00Z" }

Investigate High Token Usage

Ask: "Which requests used the most tokens today?"

Tool call: get_llm_expensive_traces

{ "limit": 10, "start_time": "2024-01-15T00:00:00Z", "min_tokens": 5000 }

Common Workflows

Cost Optimization

• get_llm_expensive_traces — find the highest-token requests
• get_llm_usage — see which models are costing the most
• get_trace on a specific trace_id — inspect the exact prompt/response

Performance Debugging

• get_llm_slow_traces — identify latency outliers
• find_errors — check for failure patterns
• get_llm_model_stats — check finish-reason distribution for truncation

Model Adoption Tracking

• list_llm_models — see every model actually being called
• get_llm_model_stats per model — compare performance
• Scan list_llm_models results for unexpected models/services (shadow AI)

Development

git clone https://github.com/mcpsmiths/tracehub-mcp.git
cd tracehub-mcp
uv sync --group dev   # pulls in pytest, mypy, ruff, etc. for local iteration

# Tests (458 passed, 2 skipped at time of writing)
uv run pytest

# With coverage
uv run pytest --cov=opentelemetry_mcp --cov-report=html

# Format, lint, type-check
uv run ruff format .
uv run ruff check .
uv run mypy src/

CI (.github/workflows/ci.yml) runs Ruff and mypy (strict) on every push, plus the full pytest suite.

Troubleshooting

Backend connection issues:

curl http://localhost:16686/api/services   # Jaeger
curl http://localhost:3200/api/search/tags  # Tempo

Authentication errors: confirm your key is set —

export BACKEND_API_KEY=your_key_here
# or: tracehub-mcp --api-key your_key_here

No traces found:

• Check the time range (use recent timestamps)
• Verify service names with list_services
• Try searching without filters first

Token usage shows zero:

• Confirm your traces have OpenTelemetry gen_ai.* (or legacy llm.*) instrumentation
• Inspect raw span attributes with get_trace

Datadog/Sentry-specific issues: see the troubleshooting notes under each backend in Configuration.

Roadmap

Both ideas that were previously listed here as deferred are now shipped:

• Agent-native triage — tools that flag a likely root cause rather than just returning raw trace data: see triage_trace in the Tools Reference above.
• Cross-backend correlation — querying a second, independently-configured backend and correlating results against it: see correlate_trace above. This ships as a best-effort, heuristic correlation (direct trace_id match, or a time-window/service-overlap fallback) rather than a guaranteed schema-level join, per the research behind it — trace-id continuity across a vendor boundary is not guaranteed even under normal W3C Trace Context propagation.

New Relic and Honeycomb have since shipped as the 7th and 8th backends, both built entirely from research/documentation without a live account to verify against — see their own setup sections for exactly what's unverified before relying on either in production; Honeycomb in particular carries a real go/no-go risk (its programmatic Query Data API is documented as Enterprise-plan exclusive). There is no further backend currently under research. (Grafana Cloud was never on this list — it already ships today via Tempo's Basic Auth path; see Grafana Tempo above.)

Carried over from upstream's older roadmap, re-prioritized behind the above rather than dropped: a dedicated model-vs-model comparison tool (today's get_llm_model_stats and compare_time_windows cover per-model stats and time-window diffing separately, but not a single tool that diffs two specific models directly), broader prompt-pattern analysis across templates rather than just version-over-version for one prompt (get_prompt_version_stats already covers the latter), MCP resources for common queries, and SigNoz/ClickHouse backend support. Cost calculation with built-in pricing tables and a query-result caching layer are shipped, not pending: every usage-reporting tool (e.g. investigate_cost_spike in the Tools Reference) returns a cost_usd/cost_usd_is_partial pair backed by a vendored litellm pricing table (src/opentelemetry_mcp/pricing/), and query results are cached with in-flight request coalescing via the QUERY_CACHE_TTL_SECONDS env var (see the configuration table above).

Everything else documented elsewhere in this README is shipped.

Contributing

Contributions are welcome. Before opening a PR, make sure:

• All tests pass: uv run pytest
• Code is formatted: uv run ruff format .
• No linting errors: uv run ruff check .
• Type checking passes: uv run mypy src/

License

Apache License 2.0 — see LICENSE. This project is a fork of traceloop/opentelemetry-mcp-server; full attribution and the fork relationship are documented in NOTICE.

Support

• Issues & feature requests: github.com/mcpsmiths/tracehub-mcp/issues
• Changelog: CHANGELOG.md
• Community: Traceloop Community Slack (upstream's community channel; general OpenTelemetry/GenAI-tracing questions welcome)
• Related: Model Context Protocol · OpenTelemetry GenAI semantic conventions · traceloop/opentelemetry-mcp-server (upstream)