pi-web-kit

pi-web-kit

Context-efficient web search and fetch tools for Pi: web_search and web_fetch.

pi-web-kit provides provider-backed search and page fetching with bounded output, chunked reads, URL validation, and an in-memory fetch cache designed for agent workflows.

Features

• web_search for current/external web information, including multi-query searches.
• web_fetch for reading one or more URLs, with offset / limit chunk reads for long pages.
• Multiple provider backends: Exa MCP, Exa API, TinyFish, Brave Search, Firecrawl, and markdown.new.
• Provider-tailored tool schemas at Pi startup/reload.
• URL validation: HTTP(S)-only, no embedded credentials, fragment stripping, duplicate removal, and length/count limits.
• In-memory fetch cache with TTL, LRU eviction, max entry count, max byte count, and cache keys based on provider/config/fetch-affecting options.
• Bounded provider concurrency and network timeouts.

Installation

Install from npm:

pi install npm:pi-web-kit

Install project-locally with Pi's -l flag:

pi install -l npm:pi-web-kit

During local development from this monorepo:

pi install /path/to/pi-mono/packages/pi-web-kit

For a one-off test run without installing:

pi -e /path/to/pi-mono/packages/pi-web-kit --web-provider-fetch markdown_new --print "Fetch https://example.com"

This is an npm-compatible TypeScript Pi package. Bun is not required.

Quick usage

Search:

Find recent documentation for the Pi extension API.

Multi-query search:

Search for recent docs on Pi extensions and Pi tool schemas.

Fetch a page:

Read https://example.com and summarize it.

Fetch a long page in chunks:

Fetch https://example.com/long-doc with limit 8000, then continue with offset 8000.

Pi chooses web_search or web_fetch automatically when the request calls for it. You can also mention provider settings explicitly in prompts, but provider changes usually require config or CLI flags.

Providers

Defaults: provider_search = "exa_mcp", provider_fetch = "exa_mcp".

Provider	Search	Fetch	Key
exa_mcp	yes	yes	optional EXA_API_KEY
exa	yes	yes	EXA_API_KEY
tinyfish	yes	yes	TINYFISH_API_KEY
brave	yes	no	BRAVE_SEARCH_API_KEY
firecrawl	yes	yes	FIRECRAWL_API_KEY
markdown_new	no	yes	none

Tool schemas are tailored to the configured providers at startup/reload, so only supported provider-specific fields are exposed. Restart/reload Pi after changing provider config.

Configuration

Resolution order: defaults < environment variables < global config < project config < CLI flags.

Environment variables

PI_OFFLINE=1        # disables install/update telemetry
PI_TELEMETRY=0      # disables install/update telemetry
PI_WEB_KIT_PROVIDER_SEARCH=exa_mcp|exa|tinyfish|brave|firecrawl
PI_WEB_KIT_PROVIDER_FETCH=exa_mcp|exa|tinyfish|markdown_new|firecrawl
EXA_API_KEY=...
TINYFISH_API_KEY=...
BRAVE_SEARCH_API_KEY=...
FIRECRAWL_API_KEY=...

Config files

Config files, in increasing precedence:

Scope	Path
Global	~/.pi/agent/pi-web-kit.json
Project	.pi-web-kit.json

Example:

{
  "provider_search": "firecrawl",
  "provider_fetch": "markdown_new",
  "apiKeys": {
    "firecrawl": "..."
  },
  "markdownNew": {
    "method": "auto",
    "retainImages": false
  }
}

Do not commit config files containing secrets. Project .pi-web-kit.json is ignored by this repo's .gitignore, but other repositories may need their own ignore rule.

CLI overrides

pi -e . --web-provider-search firecrawl --web-provider-fetch markdown_new --print "Search and fetch docs"

Provider CLI flags are temporary for the Pi process. Restart/reload Pi after changing provider config so registered tool schemas match the active provider.

Tools

web_search

Searches with the active search provider and returns compact results grouped by query.

Parameter	Type	Description
query	string	Single search query.
queries	string[]	Multiple related search queries. Max 5 after de-duplication.
numResults	integer	Results per query. Range: 1-20. Default: 10.

Provider-specific parameters are exposed only for the configured provider, such as Exa date/domain filters, TinyFish page, Brave locale/freshness options, or Firecrawl scrape/search options.

web_fetch

Fetches page content with the active fetch provider. Results are cached in memory by canonical URL plus provider/config/fetch-affecting options.

Parameter	Type	Description
url	string	Single URL. Must be http: or https:.
urls	string[]	Multiple URLs. Max 10 after de-duplication.
offset	integer	Character offset for cached/ranged reads. Single URL only.
limit	integer	Maximum characters to return. Default: 30,000 for one URL, 8,000 for multiple URLs.
refresh	boolean	Refetch even if cached.

Provider-specific parameters are exposed only for the configured provider, such as TinyFish format, markdown.new method / retainImages, or Firecrawl format, waitFor, mobile, location, and maxAge.

Cache and limits

web_fetch uses an in-memory cache for the current Pi process.

Limit	Value
Cache TTL	30 minutes
Max cached entries	100
Max cached bytes	20 MiB
Max URLs per call	10
Max queries per call	5
Max numResults	20
Max URL length	2048 characters

Cache keys include the provider, canonical URL, fetch-affecting parameters, relevant provider defaults, and a short API-key/account scope marker. refresh: true bypasses and replaces the cached entry.

Privacy and security

pi-web-kit sends search queries and fetched URLs to the configured provider. Fetch providers may also receive provider-specific options. API keys are read from environment variables or local config files and are used only for provider requests.

The extension rejects non-HTTP(S) URLs and URLs with embedded username/password credentials. Provider responses are not sandboxed; they are returned to Pi as tool output.

Report security issues privately. See SECURITY.md.

Troubleshooting

Symptom	Cause	Fix
provider requires ... API_KEY	Selected provider needs an API key.	Set the provider's env var or apiKeys config entry.
Provider mismatch / schema error after config change	Pi registered tools for the previous startup provider.	Restart/reload Pi after provider changes.
Invalid URL / scheme / credentials error	URL validation rejected the input.	Use an absolute http: or https: URL without username/password credentials.
Timeout error	Provider request exceeded its timeout.	Retry, reduce URL count, or switch provider.
No content returned	Provider returned no matching content or a redirected/canonicalized response could not be mapped.	Retry with refresh: true, fetch a single URL, or switch provider.
Large page is truncated	Tool output is bounded for context efficiency.	Use offset and limit to continue reading chunks.

Development

Requirements:

• Node.js >= 20.6.0
• npm

Common commands:

npm install
npm run check
npm test
npm audit --omit=dev
npm run pack:dry-run

This package is source-distributed. Pi loads the TypeScript extension files directly via its extension loader.

Contributing

Contributions are welcome. See CONTRIBUTING.md for development workflow and pull request guidelines.

License

MIT. See LICENSE.