@coctostan/pi-exa-gh-web-tools
Advanced tools
Web search via Exa, content extraction, and GitHub repo cloning for Pi coding agent
Web search, code search, content extraction, and GitHub repo cloning for the Pi coding agent, powered by Exa.
This package gives Pi four tools:
web_search — search the web and return compact resultscode_search — find code examples from docs, GitHub, and Stack Overflowfetch_content — fetch a URL, GitHub repo/file, or PDF and extract readable contentget_search_content — retrieve stored content from an earlier tool callMost web pages are too large and noisy to drop directly into an agent's context window. This extension is designed to keep Pi focused:
web_search returns short summaries by defaultfetch_content can answer a specific question instead of returning a whole pageIf you're new to Pi, the simplest mental model is:
0.74.0 (npm scope @earendil-works/*)pi-web-tools declares peerDependencies on @earendil-works/pi-coding-agent ^0.74.0 and @earendil-works/pi-tui ^0.74.0. The legacy @mariozechner/* scope is no longer supported — if you are on pi < 0.74, stay on pi-web-tools@3.x.
From npm:
Requires pi 0.74 or newer.
pi install npm:@coctostan/pi-exa-gh-web-tools
Or directly from GitHub:
Requires pi 0.74 or newer.
pi install github:coctostan/pi-web-tools
web_search and code_search require an Exa API key.
Set it as an environment variable:
export EXA_API_KEY="your-key-here"
Or put it in ~/.pi/web-tools.json:
{
"exaApiKey": "your-key-here"
}
Environment variables take precedence over the config file.
Typical beginner flow:
web_search({ query: "vitest mock fetch" })
fetch_content({
url: "https://vitest.dev/guide/mocking.html",
prompt: "How do I mock a function in Vitest?"
})
The package also ships a standalone exa-tools binary that works outside of Pi.
npm install -g @coctostan/pi-exa-gh-web-tools
search and code commands require an Exa API key:
export EXA_API_KEY="your-key-here"
Web search:
exa-tools search "vitest mock fetch" --n 3
Code search:
exa-tools code "vitest mock fetch" --tokens 800
Fetch a page (raw markdown):
exa-tools fetch "https://vitest.dev/guide/mocking.html"
Fetch with a focused question:
exa-tools fetch "https://vitest.dev/guide/mocking.html" --prompt "How do I mock a function?"
--prompt is used but no filter model is available, the CLI prints a warning to stderr and falls back to raw markdown on stdoutIf you've never used Pi tools before, this is the shortest useful workflow:
// 1) Find a good source
web_search({ query: "vitest retry failed test" })
// 2) Ask one page a focused question
fetch_content({
url: "https://vitest.dev/guide/",
prompt: "How do I retry a failed test?"
})
Rule of thumb:
web_search to choose a sourcefetch_content({ prompt }) to get an answerfetch_content({ url }) without prompt only when you really need the raw page| If you want to... | Use this |
|---|---|
| Find a relevant page or article | web_search |
| Find a working code snippet | code_search |
| Ask one URL a specific question | fetch_content({ url, prompt }) |
| Read the full raw content of a page | fetch_content({ url }) |
| Re-open an earlier result without refetching | get_search_content |
For most Pi sessions, this is the best default path:
web_searchfetch_content({ prompt })get_search_content or read only if you need more detailweb_searchSearch the web and return 1-line summaries by default.
Use it when you want to decide which URL is worth reading next.
| Parameter | Type | Description |
|---|---|---|
query | string | Single search query |
queries | string[] | Multiple search queries |
numResults | number | Results per query, default 5, max 20 |
type | string | "auto" (default), "instant", or "deep" |
detail | string | "summary" (default) or "highlights" |
freshness | string | "realtime" (last 1 hour), "day" (24h), "week" (168h), or "any" (no freshness filter) |
category | string | Content category filter |
includeDomains | string[] | Only include these domains |
excludeDomains | string[] | Exclude these domains |
similarUrl | string | Find pages similar to a URL |
// Basic search
web_search({ query: "vitest snapshot testing" })
// Get more detail before fetching
web_search({ query: "rust async runtime comparison", detail: "highlights" })
// Restrict results to specific sites
web_search({ query: "useEffect cleanup", includeDomains: ["react.dev", "github.com"] })
// Batch search
web_search({ queries: ["vitest mocking", "vitest coverage", "vitest browser mode"] })
// Find related pages
web_search({ similarUrl: "https://vitest.dev/guide/" })
The tool automatically improves certain queries before sending them to Exa:
docs examplefetch_contentFetch a page, GitHub repo/file, or PDF and return readable content.
Use it when you already know which source you want to inspect.
| Parameter | Type | Description |
|---|---|---|
url | string | Single URL to fetch |
urls | string[] | Multiple URLs to fetch |
prompt | string | Ask a question about the content instead of returning the whole page |
forceClone | boolean | Force clone for large GitHub repos |
noCache | boolean | Skip research cache and fetch fresh (still updates cache) |
Prefer prompt whenever you can.
fetch_content({
url: "https://vitest.dev/guide/",
prompt: "How do I run only one test file?"
})
That returns a focused answer instead of dumping a large page into context.
Without prompt, content is written to a temp file and the tool returns:
This keeps Pi's context smaller while preserving access to the full content.
GitHub URLs are detected automatically.
// Repo tree + README summary
fetch_content({ url: "https://github.com/facebook/react" })
// Specific file
fetch_content({ url: "https://github.com/facebook/react/blob/main/packages/react/src/React.js" })
The tool tries gh repo clone first, then falls back to git clone.
fetch_content({ url: "https://arxiv.org/pdf/2312.00752" })
PDF text is extracted with unpdf (a serverless build of Mozilla pdf.js). Corrupt, encrypted, empty, or oversized PDFs return a clear error.
code_searchSearch for working code examples from docs, GitHub repositories, and Stack Overflow.
Use it when you want code patterns, not general web pages.
| Parameter | Type | Description |
|---|---|---|
query | string | Describe what code you want |
tokensNum | number | Response size in tokens |
code_search({ query: "vitest mock fetch with MSW" })
code_search({ query: "React Server Components with Next.js app router", tokensNum: 5000 })
get_search_contentRetrieve stored content from an earlier web_search, fetch_content, or code_search call.
This is useful when you want to revisit a result without repeating the network request.
| Parameter | Type | Description |
|---|---|---|
responseId | string | ID returned by an earlier tool call |
query | string | Retrieve a web_search result by query |
queryIndex | number | Retrieve a web_search result by position |
url | string | Retrieve a fetch_content result by URL |
urlIndex | number | Retrieve a fetch_content result by position |
maxChars | number | Maximum response size, default 30000, max 100000 |
get_search_content({ responseId: "abc123", queryIndex: 0 })
get_search_content({ responseId: "xyz789", url: "https://vitest.dev/api/" })
The package reads config from ~/.pi/web-tools.json and hot-reloads it every 30 seconds.
To choose the model used by fetch_content({ prompt }), add filterModel to that file using provider/model-id format:
{
"filterModel": "openai-codex/gpt-5.4-mini"
}
You can point pi-web-tools at another config file with PI_WEB_TOOLS_CONFIG=/path/to/web-tools.json. If filterModel is omitted or malformed, pi-web-tools treats it as unset and uses auto-detection.
{
"exaApiKey": "your-exa-key",
"filterModel": "openai-codex/gpt-5.4-mini",
"cacheTTLMinutes": 1440,
"github": {
"maxRepoSizeMB": 350,
"cloneTimeoutSeconds": 30,
"clonePath": "/tmp/pi-github-repos"
},
"tools": {
"web_search": true,
"code_search": true,
"fetch_content": true,
"get_search_content": true
}
}
| Setting | Description |
|---|---|
exaApiKey | Exa API key used by web_search and code_search |
filterModel | Summarization/filter model used by fetch_content({ prompt }), in provider/model-id format |
github.maxRepoSizeMB | Max GitHub repo size before refusing or requiring force clone |
github.cloneTimeoutSeconds | Clone timeout |
github.clonePath | Cache directory for cloned repos |
tools.* | Enable or disable individual tools |
cacheTTLMinutes | TTL in minutes for the persistent research cache (default: 1440 = 24h) |
Omit filterModel to let pi-web-tools auto-detect an available cheap filter model from its built-in candidate list. The config field is intentionally named filterModel; there is no separate summarizationModel setting.
To use a different config path:
export PI_WEB_TOOLS_CONFIG="$HOME/.pi/web-tools.json"
This package is opinionated about token efficiency.
web_search returns short summaries by default so the main model only sees enough to choose a source.
fetch_content({ prompt }) lets a cheaper model read the full page and return only the answer to your question.
Raw fetched content is offloaded to a temp file instead of being pasted inline.
Search and fetch results stay available for the session through get_search_content.
All Exa API requests use retry logic for transient failures.
1s -> 2s429, 500, 502, 503, 504, and network errors400, 401, 403, 404, and abort signalsThe package also:
p-limit(3)p-limit(3)Clone the repo:
git clone git@github.com:coctostan/pi-web-tools.git
cd pi-web-tools
npm install
Run tests:
npm test
Watch tests while developing:
npm run test:watch
Load the extension in Pi for manual testing:
pi -e ./index.ts
The repo vendors a minimal node_modules/ snapshot under .pi/npm/ so that
pi -e ./index.ts runs against a pinned coding-agent build. To refresh it
after a pi release:
rm -rf .pi/npm/node_modules .pi/npm/package-lock.json
(cd .pi/npm && npm install)
npx tsx scripts/smoke-load-extension.mjs
Commit the resulting .pi/npm/package.json, .pi/npm/package-lock.json, and
.pi/npm/node_modules/ tree.
Tests use mocked network calls, so they do not require an Exa API key.
web_search or code_search fails immediatelyUsually this means your Exa API key is missing or invalid.
Check:
echo "$EXA_API_KEY"
Or verify ~/.pi/web-tools.json contains:
{
"exaApiKey": "your-key-here"
}
fetch_content returned a file path instead of an answerThat is expected when you do not provide prompt, or when no cheap filter model is available.
Use:
fetch_content({
url: "https://example.com",
prompt: "What does this page say about X?"
})
Try this order:
web_search firstfetch_content({ prompt }) instead of raw fetchTry:
fetch_content({
url: "https://github.com/owner/repo",
forceClone: true
})
Also make sure gh or git is available on your machine.
The repo is currently at package version 2.0.0. If npm still shows an older version, use this checklist before publishing:
npm test
npm pack --dry-run
npm publish --access public
Before publishing, confirm:
package.json version is correctREADME.md reflects the current feature setCurrent package metadata points to this repo:
https://github.com/coctostan/pi-web-toolshttps://github.com/coctostan/pi-web-tools/issueshttps://github.com/coctostan/pi-web-tools#readmeindex.ts Pi extension entry point and tool registration
exa-search.ts Exa web search integration
exa-context.ts Exa code/context search integration
extract.ts HTML/PDF content extraction
github-extract.ts GitHub repo and file handling
filter.ts Cheap-model filtering for focused answers
research-cache.ts Persistent TTL-based research cache
storage.ts Session result storage
config.ts Config loading and hot reload
tool-params.ts Tool input normalization and validation
retry.ts Retry and backoff helpers
offload.ts Temp-file offload for raw content
smart-search.ts Query enhancement and deduplication
truncation.ts Response truncation helpers
constants.ts Shared constants (timeouts, TTLs)
signal directly to Exa/extract/filter calls; the manual pendingFetches Map and abortAllPending helper are gone (~30 lines removed per tool).session_start lifecycle: branch on event.reason — reload preserves the URL cache and temp files, new starts clean, fork restores from event.previousSessionFile via the new restoreFromSessionFile helper.prepareArguments adoption: all four tools (web_search, fetch_content, code_search, get_search_content) wire their normalize*Input functions into pi's ToolDefinition.prepareArguments hook. numResults is now a bounded integer in the visible schema.get_search_content no longer fails after /compact. The session result store is mirrored to ~/.pi/cache/web-tools/results-<sessionId>.json and rehydrated on session_start. Files older than 24h are pruned automatically.0.74.0+ and the @earendil-works/* npm scope. Users on older pi must stay on pi-web-tools@3.x.ModelRegistry.getApiKeyAndHeaders and threads custom auth headers (Anthropic OAuth, Cloudflare AI Gateway, Xiaomi) through to the filter modelsession_start{reason} event; reload no longer wipes the URL cache or temp files.pi/npm extension snapshot to @earendil-works/pi-coding-agent@^0.74.0fetch_content gained persistent research cache — repeated prompt+URL lookups return instant cached answersfetch_content gained noCache param to bypass cachecacheTTLMinutes config option (default 24h)details.ptcValue on all 4 tools for PTC interopfetch_content gained prompt for focused question answeringweb_search now returns summary-first results by defaultweb_search gained freshness, similarUrl, and detailfetch_contentget_search_content.maxCharsweb_search, code_search, fetch_content, and get_search_contentThis project is licensed under the MIT License. See LICENSE.
FAQs
Web search via Exa, content extraction, and GitHub repo cloning for Pi coding agent
The npm package @coctostan/pi-exa-gh-web-tools receives a total of 8 weekly downloads. As such, @coctostan/pi-exa-gh-web-tools popularity was classified as not popular.
We found that @coctostan/pi-exa-gh-web-tools demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Threat Research
/Security News
Socket uncovered secret-stealing workflows that ran in LiteLLM’s GitHub repository 11 days before the supply chain attack on PyPI and Docker.
Security News
Cloudflare introduces tsrs, a Rust port of TypeScript’s compiler promising faster checks and lower memory use, with the TypeScript team’s help.
Security News
The PHP Foundation says Chainguard broke a security embargo ahead of critical RCE patches, as leaks disrupted the coordinated release.