mdrip - npm Package Compare versions

Comparing version 0.1.3 to 0.1.5

dist/index.js

@@ -11,3 +11,3 @@ #!/usr/bin/env node
     .description("Fetch markdown snapshots for URLs using Cloudflare Markdown for Agents")
-    .version("0.1.3")
+    .version("0.1.4")
     .option("--cwd <path>", "working directory (default: current directory)");
@@ -14,0 +14,0 @@ program

package.json

 {
   "name": "mdrip",
-  "version": "0.1.3",
+  "version": "0.1.5",
   "description": "Fetch markdown snapshots of web pages using Cloudflare Markdown for Agents",
@@ -41,2 +41,3 @@ "type": "module",
     "start": "node dist/index.js",
+    "benchmark": "node scripts/benchmark.mjs",
     "test": "vitest run",
@@ -43,0 +44,0 @@ "test:watch": "vitest",

README.md

 # mdrip
 
-Fetch markdown snapshots of web pages using Cloudflare's Markdown for Agents feature, so coding agents can consume clean structured content instead of HTML.
+Fetch clean markdown snapshots of any web page — optimized for AI agents, RAG pipelines, and context-aware workflows.
 
-## AI Skills
+Reduces token overhead by ~90% compared to raw HTML while preserving the content structure LLMs need.
 
-This repo also includes an AI-consumable skills catalog in `skills/`, following the [agentskills](https://agentskills.io) format.
+## Why
 
-- Skill index: `skills/README.md`
-- mdrip skill: `skills/mdrip/SKILL.md`
+AI agents and LLMs work better with markdown than HTML. Feeding raw HTML into a context window wastes tokens on tags, scripts, styles, and boilerplate. mdrip solves this by fetching any URL and returning clean, structured markdown.
 
-### Install skills from this repo
+- **~90% fewer tokens** than raw HTML
+- **Automatic HTML-to-markdown fallback** when native markdown isn't available
+- **Works everywhere** — CLI, Node.js, Cloudflare Workers, or via remote MCP
+- **Token-aware** — reports estimated token counts so you can manage context budgets
 
-If you use a Skills-compatible agent setup, you can add these skills directly:
+Sites that support [Cloudflare's Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) return markdown natively at the edge. For all other sites, mdrip's built-in converter handles headings, links, lists, code blocks, tables, blockquotes, and more.
 
+## Installation
+
 ```bash
-# install skills from this repo
-npx skills add charl-kruger/mdrip
+npm install -g mdrip
 ```
 
-## Why
+Or use directly with `npx`:
 
-For agent workflows, markdown is often better than HTML:
-- cleaner structure
-- lower token overhead
-- easier chunking and context management
+```bash
+npx mdrip <url>
+```
 
-`mdrip` requests pages with `Accept: text/markdown`, stores the markdown locally, and tracks fetched pages in an index.
+## CLI Usage
 
-If a site does not return `text/markdown`, `mdrip` can automatically fall back to converting `text/html` into markdown.
-The fallback uses an in-project converter optimized for common documentation/blog content (headings, links, lists, code blocks, tables, blockquotes).
+### Fetch pages
 
-## Why Cloudflare Markdown for Agents matters
+```bash
+# Fetch one page
+mdrip https://example.com/docs/getting-started
 
-Cloudflare's blog and docs describe Markdown for Agents as content negotiation at the edge:
-- clients request `Accept: text/markdown`
-- Cloudflare converts HTML to markdown in real time (for enabled zones)
-- response includes `x-markdown-tokens` for token-size awareness
+# Fetch multiple pages
+mdrip https://example.com/docs https://example.com/api
 
-For AI workflows this is high-value:
-- better structure for LLM parsing than raw HTML
-- less token waste in context windows
-- predictable markdown snapshots you can store and reuse in your repo
+# Custom timeout (ms)
+mdrip https://example.com --timeout 45000
 
-References:
-- [Cloudflare blog: Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/)
-- [Cloudflare docs: Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/)
+# Strict mode — only accept native markdown, no HTML fallback
+mdrip https://example.com --no-html-fallback
 
-## Installation
+# Raw mode — print markdown to stdout, no file writes
+mdrip https://example.com --raw
+```
 
+### List fetched pages
+
 ```bash
-npm install -g mdrip
+mdrip list
+mdrip list --json
 ```
 
-Or use with `npx`:
+### Remove pages
 
 ```bash
-npx mdrip <url>
+mdrip remove https://example.com/docs/getting-started
 ```
 
-For programmatic usage in Node.js or Workers:
+### Clean snapshots
 
 ```bash
-npm install mdrip
+# Remove all
+mdrip clean
+
+# Remove only one domain
+mdrip clean --domain example.com
 ```
 
-## Programmatic API
+### Raw mode for agent runtimes
 
-### Node.js (fetch and store)
+`--raw` prints markdown to stdout and skips all file writes and prompts. Useful for piping content directly into agent loops.
 
-```ts
-import { fetchToStore, listStoredPages } from "mdrip/node";
+```bash
+mdrip https://example.com --raw | your-agent-cli
+```
 
-const result = await fetchToStore("https://developers.cloudflare.com/", {
-  cwd: process.cwd(),
-});
+## Programmatic API
 
-if (!result.success) {
-  throw new Error(result.error || "Failed to fetch page");
-}
-
-const pages = await listStoredPages(process.cwd());
-console.log(pages.map((p) => p.path));
+```bash
+npm install mdrip
 ```
 
-### Cloudflare Workers / Agent runtimes (raw in-memory markdown)
+### Workers / Edge / In-memory
 
@@ -91,94 +93,75 @@ ```ts
 
-const page = await fetchMarkdown(
-  "https://blog.cloudflare.com/markdown-for-agents/",
-);
+const page = await fetchMarkdown("https://example.com/docs");
 
-console.log(page.markdownTokens);
-console.log(page.markdown);
+console.log(page.markdown);       // clean markdown content
+console.log(page.markdownTokens); // estimated token count
+console.log(page.source);         // "cloudflare-markdown" or "html-fallback"
 ```
 
-Available programmatic methods:
-- `mdrip` (Workers-safe): `fetchMarkdown(url, options)`, `fetchRawMarkdown(url, options)`
-- `mdrip/node` (filesystem features): `fetchToStore(url, options)`, `fetchManyToStore(urls, options)`, `listStoredPages(cwd?)`
+### Node.js (fetch and store to disk)
 
-## Usage
+```ts
+import { fetchToStore, listStoredPages } from "mdrip/node";
 
-### Fetch pages
+const result = await fetchToStore("https://example.com/docs", {
+  cwd: process.cwd(),
+});
 
-```bash
-# Fetch one page
-mdrip https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/
+if (result.success) {
+  console.log(`Saved to ${result.path}`);
+}
 
-# Fetch multiple pages
-mdrip https://blog.cloudflare.com/markdown-for-agents/ https://developers.cloudflare.com/
+const pages = await listStoredPages(process.cwd());
+```
 
-# Optional timeout override (ms)
-mdrip https://example.com --timeout 45000
+### Available exports
 
-# Disable HTML fallback (strict Cloudflare markdown only)
-mdrip https://example.com --no-html-fallback
+| Import | Environment | Functions |
+|--------|-------------|-----------|
+| `mdrip` | Workers, edge, browser | `fetchMarkdown()`, `fetchRawMarkdown()` |
+| `mdrip/node` | Node.js | `fetchToStore()`, `fetchManyToStore()`, `listStoredPages()` |
 
-# Print raw page markdown to stdout (no files/settings changes, no prompts)
-mdrip https://blog.cloudflare.com/markdown-for-agents/ --raw
-```
+## Remote MCP Server
 
-### Raw mode for agents (OpenClaw, etc.)
+mdrip is available as a remote MCP server at **`mdrip.createmcp.dev`** — no install required. Any MCP-compatible client can connect and use the `fetch_markdown` and `batch_fetch_markdown` tools.
 
-`--raw` is designed for agent runtimes that only need in-memory content.
-It prints markdown to stdout and skips settings prompts and all file writes.
+### Claude Desktop
 
-This is useful for flows with OpenClaw and similar AI tools where you want to pipe page content directly into your agent loop.
+Add to `claude_desktop_config.json`:
 
-```bash
-# stream markdown directly to another process
-mdrip https://blog.cloudflare.com/markdown-for-agents/ --raw
+```json
+{
+  "mcpServers": {
+    "mdrip": {
+      "command": "npx",
+      "args": ["mcp-remote", "https://mdrip.createmcp.dev/mcp"]
+    }
+  }
+}
 ```
 
-### List fetched pages
+### Claude Code
 
 ```bash
-mdrip list
-mdrip list --json
+claude mcp add mdrip-remote --transport sse https://mdrip.createmcp.dev/sse
 ```
 
-### Remove pages
+### Cloudflare AI Playground
 
-```bash
-mdrip remove https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/
-```
+Enter `mdrip.createmcp.dev/sse` at [playground.ai.cloudflare.com](https://playground.ai.cloudflare.com/).
 
-### Clean snapshots
-
-```bash
-# Remove all
-mdrip clean
-
-# Remove only one domain
-mdrip clean --domain developers.cloudflare.com
-```
-
 ## File modifications
 
 On first run, mdrip can optionally update:
-- `.gitignore` (adds `mdrip/`)
-- `tsconfig.json` (excludes `mdrip`)
-- `AGENTS.md` (adds a section pointing agents to snapshots)
+- `.gitignore` — adds `mdrip/`
+- `tsconfig.json` — excludes `mdrip/`
+- `AGENTS.md` — adds a section pointing agents to your snapshots
 
-Choice is stored in `mdrip/settings.json`.
+Choice is stored in `mdrip/settings.json`. Use `--modify` or `--modify=false` to skip the prompt.
 
-Use flags to skip prompt:
+`--raw` mode bypasses this entirely.
 
-```bash
-# allow updates
-mdrip https://example.com --modify
+## Output structure
 
-# deny updates
-mdrip https://example.com --modify=false
 ```
-
-`--raw` mode bypasses this entire flow and never writes settings or snapshots.
-
-## Output
-
-```text
 mdrip/
@@ -188,30 +171,36 @@ ├── settings.json
 └── pages/
-    └── developers.cloudflare.com/
-        └── fundamentals/
-            └── reference/
-                └── markdown-for-agents/
-                    └── index.md
+    └── example.com/
+        └── docs/
+            └── getting-started/
+                └── index.md
 ```
 
-## Requirements and notes
+## Benchmark
 
-- Node.js 18+
-- The target site must return markdown for `Accept: text/markdown` (Cloudflare Markdown for Agents enabled).
-- If a page does not return `text/markdown`, mdrip can convert `text/html` into markdown fallback unless `--no-html-fallback` is used.
+Measured across popular pages (values vary as pages change):
 
-## Publishing to npm
+| Page | Mode | Chars saved | Tokens saved |
+|------|------|------------:|-------------:|
+| blog.cloudflare.com/markdown-for-agents | cloudflare-markdown | 94.9% | 94.9% |
+| developers.cloudflare.com/.../markdown-for-agents | cloudflare-markdown | 95.7% | 95.7% |
+| en.wikipedia.org/wiki/Markdown | html-fallback | 72.7% | 72.7% |
+| github.com/cloudflare/skills | html-fallback | 96.3% | 96.3% |
+| **Average** | | **89.9%** | **89.9%** |
 
 ```bash
-# optional package check
-pnpm publish:dry-run
+pnpm build && pnpm benchmark
+```
 
-# publish to npm
-pnpm publish:npm
+## AI Skills
+
+This repo includes an AI-consumable skills catalog in `skills/`, following the [agentskills](https://agentskills.io) format.
+
+```bash
+npx skills add charl-kruger/mdrip
 ```
 
-`prepublishOnly` runs automatically before publish and executes:
-- `pnpm type-check`
-- `pnpm test`
-- `pnpm build`
+## Requirements
 
+- Node.js 18+
+
 ## Author
@@ -218,0 +207,0 @@

New alerts

URL strings

Supply chain risk

Package contains fragments of external URLs or IP addresses, which the package may be accessing at runtime.

Fixed alerts

URL strings

Supply chain risk

Package contains fragments of external URLs or IP addresses, which the package may be accessing at runtime.

Worsened metrics

Total package byte prevSize

141605

-0.29%

Number of lines in readme file

211

-4.95%

No dependency changes