@sanity/agent-context-explorer

Agent Context Explorer

Companion tool for Sanity Agent Context. Explores your Agent Context server, documents what works and what doesn't, and produces knowledge docs (dataset-knowledge.md) that help production agents operate effectively from day one.

Why This Tool?

When building AI agents that work with Agent Context, agents need to know:

• What data exists and what's reliably populated
• How to query effectively for different question types
• What caveats and hazards will trip them up (the most valuable knowledge)

Manual exploration is time-consuming and doesn't scale. This tool automates that process, producing structured documentation that helps production agents work correctly from day one.

What You Get

After running the explorer, you'll have documentation that tells agents:

• Which document types to query (and which to ignore)
• Working query patterns for different question types
• Critical hazards — data gaps, external system dependencies, and silent failure modes that would otherwise cause wrong answers

Installation

# Global install (recommended — reuse across projects)
npm install -g @sanity/agent-context-explorer

# Or install per-project
npm install @sanity/agent-context-explorer

Quick Start

• Create a questions file (questions.json). Always include expected_answer — the explorer uses it to guide exploration and validate results:

{
  "questions": [
    { "question": "What sizes does the Trailblazer Hiking Boot come in?", "expected_answer": "US 7-13, including half sizes" },
    { "question": "Is the Ultralight Tent waterproof?", "expected_answer": "Yes, it has a 3000mm waterproof rating with taped seams" },
    { "question": "What's the difference between the Summit Pack and the Daybreak Pack?", "expected_answer": "Summit is 65L for multi-day trips with a frame; Daybreak is 28L for day hikes" }
  ]
}

• Run the explorer:

agent-context-explorer \
  --mcp-url https://your-mcp-server.com/mcp \
  --questions ./questions.json \
  --auth-token $MCP_AUTH_TOKEN \
  --anthropic-api-key $ANTHROPIC_API_KEY \
  --verbose

• Review the generated dataset-knowledge.md in your timestamped output directory (e.g., ./explorer-output-2026-02-11T09-22-30/).

Question File Format

{
  "questions": [
    {
      "question": string,          // The question to explore (required)
      "expected_answer"?: string,  // Strongly recommended — guides exploration and validates results
      "id"?: string                // Auto-generated as q1, q2, ... if omitted
    }
  ]
}

Always include expected_answer. The explorer uses it in two ways:

• Guides exploration — injected as a hint so the agent knows what to look for
• Validates results — the agent's answer is graded against yours, and this flows into confidence levels in the output

Without expected answers, the explorer can't tell if it found the right data. Even a rough or partial answer is better than none.

Tips for writing good questions:

• Always provide expected_answer — even approximate answers help the explorer validate findings
• Cover different question types (specs, compatibility, comparisons, troubleshooting)
• Include questions you expect to succeed AND ones that might fail
• 20-50 questions is typically sufficient for good coverage. More questions = richer documentation

Output Files

The tool generates several files in the output directory:

dataset-knowledge.md (Primary Output)

A structured knowledge document with four sections:

• What This Dataset Contains — Document types, reliable fields per type, and what is NOT in the dataset (data gaps, external dependencies)
• How to Query (by Category) — Self-contained query patterns per question category, with inline hazards and known failures
• Traps (Global Hazards) — Cross-cutting issues that affect multiple categories (field naming, query engine quirks, locale filtering)
• Trust Boundaries — What was explored vs. unknown territory, with confidence levels

Failures are the most valuable output — they document everything that would trip up a naive agent, preventing wrong answers and wasted queries.

This is the file you give to production agents.

exploration-summary.md

Human-readable summary of exploration results, useful for debugging.

logs/*.json

Individual exploration logs for each question, containing:

• All tool calls made
• Learnings and caveats discovered
• Validation results against expected answers

metrics.json

Aggregated statistics: success rates, confidence distribution, category coverage.

CLI Options

Option	Description	Default
--mcp-url <url>	Agent Context server URL (required)	—
--questions <path>	Path to questions JSON file (required)	—
--auth-token <token>	Bearer token for MCP authentication	—
--anthropic-api-key <key>	Anthropic API key (or set ANTHROPIC_API_KEY env var)	—
--model <model>	Claude model for exploration	claude-sonnet-4-20250514
--output <dir>	Output directory	./explorer-output-{timestamp}
--concurrency <n>	Number of questions to explore in parallel	5
--verbose	Enable verbose logging	false
--help	Show help message	—

How It Works

• Connect — Establishes connection to the Agent Context server and discovers available tools
• Explore — For each question, an AI agent attempts to answer it using the available tools
• Learn — The agent documents what worked, what failed, and what surprised it
• Synthesize — All exploration logs are synthesized into a unified knowledge document

The key insight: failures are more valuable than successes. A naive agent can figure out what works through trial-and-error. What they can't discover is why something that looks right doesn't work, or why data that should exist is missing.

Example Output

Here's a snippet from a generated knowledge document:

## 1. What This Dataset Contains

### What is NOT in this dataset
| Field(s) | Where It Actually Lives | What Happens If You Query Here |
|----------|------------------------|-------------------------------|
| inventory, stockLevel | Shopify | Returns null — stock managed externally |
| pricing | Commerce API | Returns null — prices not in this dataset |

## 2. How to Query (by Category)

### Category: product-specs
- **When to use**: User asks about product features, dimensions, or specifications
- **Recommended pattern**: `*[_type == "product" && name match "Trailblazer"][0]{name, specs, variants}`
- **What to avoid**: Do NOT use `title` — products use `name` (`title` is null). Do NOT query `inventory` — it is always null; stock is managed in Shopify.

## 3. Traps (Global Hazards)
| Trap | Impact | Resolution |
|------|--------|------------|
| `title` is null on all products | Queries return empty results | Use `name` field instead |
| Locale duplicates | Results contain de, fr, etc. variants | Filter by `lang == "en-us"` |

Agent Skills

After running the explorer, use the optimize-agent-prompt skill to integrate dataset-knowledge.md into your production agent's system prompt. The skill lives in the sanity-io/agent-context repository alongside the build skill:

npx skills add sanity-io/agent-context

This installs two Vercel Agent Skills:

• create-agent-with-sanity-context — Build an agent from scratch (Studio setup, agent implementation, conversation classification)
• optimize-agent-prompt — Integrate dataset-knowledge.md findings into a production-quality system prompt

Requirements

• Node.js 20+
• An Anthropic API key (pass via --anthropic-api-key or set ANTHROPIC_API_KEY env var)
• A Sanity Agent Context server URL (see Agent Context setup)

License

MIT