dsh-web-fetch-playwright

dsh-web-fetch-playwright

中文 · npm · GitHub

A DeepSeek Harness (DSH) plugin that gives the built-in web_fetch tool a Playwright/CDP backend: pages are rendered in a real browser, denoised with Readability + DOMPurify + Turndown + GFM, and returned as Markdown.

Features

• Real browser rendering — loads the page the way a user sees it, so client-side rendered (SPA) content is captured, not just the raw HTML.
• Denoise pipeline — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped tool-web renderer. Inline data: images (build tools like Docusaurus embed screenshots as base64) are elided to size placeholders such as ![alt](data:image/png;base64,...8.9KB) so they cannot flood the body.
• Two backends — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
• Browser resolution — a configured path, a playwright CLI on $PATH, or the bundled playwright-core; CDP needs no local browser at all.
• Isolated or profile sessions (CDP) — every fetch is scoped to exactly one tab. Local launches close their browser per fetch; the CDP backend keeps one shared connection to the remote browser and each fetch opens a tab inside it, closed when done. By default that tab lives in the remote browser's real profile (its cookies, localStorage, and persistent logins apply — like playwright-cli open); unchecking Share the browser context switches to a throwaway isolated context per fetch.
• Live configuration — the Plugins page (侧边栏 → 插件 → dsh-web-fetch-playwright, dsh ≥ 0.1.7) shows the full form right on the plugin's package page (and on the row's own page via its configure control); it edits the backend, context mode, denoise toggle, and concurrency, and a committed change reaches the next fetch without a restart.
• Budget-aware — per-fetch deadline (45s); concurrency is backend-priced (maxConcurrency, default 4 local browsers / 50 CDP tabs; queued fetches fail fast with a retry hint after 20s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.
• Bounded Cloudflare-challenge wait — when a navigation lands on a challenge interstitial ("Just a moment…" and its localized siblings, recognized via the documented cf-mitigated: challenge response header plus structural page markers), the fetch keeps the same tab and context and waits for the browser's own verification to clear it — tracking the last main-frame response (the real page reloads in) and watching the live DOM so SPA-style clears are caught too. Bounded and configurable (challengeWaitMs, default 15s; 0 restores the legacy first-response behavior), with a bounded same-tab retry (challengeRetries, default 1). When the budget runs out, the fetch fails with the distinct WEB_FETCH_CHALLENGE error code instead of returning the interstitial as content. It never clicks, never injects CAPTCHA answers, never fakes browser state, and never exports or copies cookies.

How it works

Half	Location	Responsibility
Host (server)	src/	Registers the fetch provider (id playwright) into ctx.web; cordis.patch.yml pins the web seam's fetchProvider to it and enables the web_fetch tool with a 60s budget.
Browser (client)	src/client/	Registers the Playwright web fetch configuration form on the Plugins page's plugins.bundle.config / plugins.row.config seats, which stages edits and writes them through the shared settings form service.

web_fetch (tool-web)
   └─ ctx.web.fetchProvider = playwright
        ├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
        ├─ cdp:   connectOverCDP(endpoint)
        ├─ page.goto → settle (networkidle, best-effort) → page.content()
        ├─ denoise: jsdom → elide data-URI images → Readability → DOMPurify → Turndown(GFM)
        └─ Markdown (or raw HTML when denoise is off)

Non-2xx statuses are results, not errors — the web seam's contract (@deepseek-ai/dsh-web): a fetched 404/418/500 comes back with its status code, exactly what dsh-web-fetch-http does. Chromium complicates this for status-only responses — an error status with an empty body (httpbin's /status/404) fails the whole navigation with net::ERR_HTTP_RESPONSE_CODE_FAILURE, and a 204/205 is aborted (net::ERR_ABORTED) because nothing is renderable. The provider therefore tracks the last main-frame response on every fetch (that response arrives before Chromium fails the navigation) and recovers such failures into the status-bearing result, so web_fetch reports Fetched … (HTTP 404) instead of a WEB_PROVIDER_ERROR throw. The same recovery covers download responses — a body served with Content-Disposition: attachment (publish.twitter.com's oembed endpoint, the URL agents use to read a tweet, serves its JSON exactly that way) never commits a document: goto rejects with Download is starting and the body is gone from the network stack, so the provider refetches it through the page's API request context and returns the decodable content; undecodable downloads (a PDF, a ZIP) still fail with WEB_UNSUPPORTED_CONTENT_TYPE. A recovered response that is itself a Cloudflare challenge edge still fails with WEB_FETCH_CHALLENGE; transport errors and cancellations stay errors.

Requirements

• DSH web profile (dsh web), Node.js ≥ 20.
• For the local backend: a Playwright installation with Chromium, a Chromium-family browser binary, or playwright-core with a browser in the default cache.
• For the CDP backend: any browser already running with --remote-debugging-port (e.g. chromium --headless --remote-debugging-port=9222).

Installation

From the npm registry (prebuilt — no build permission needed):

dsh plugin --profile web add dsh-web-fetch-playwright

From a GitHub repository (source — pnpm runs the prepare build; allowlist the package in profiles/web/pnpm-workspace.yaml if pnpm blocks the build script):

dsh plugin --profile web add github:chendefine/dsh-web-fetch-playwright

Or through the DSH plugin marketplace (设置 → DSH插件市场) — the repo carries the dsh-plugin topic and is indexed automatically.

After a bundle plugin is added to the profile layer stack, restart dsh web for it to load; uninstall with dsh plugin --profile web remove dsh-web-fetch-playwright and restart again.

Configuration

The Plugins page (侧边栏 → 插件) opens the form one click from the list: the dsh-web-fetch-playwright card lands on the package page with the full form between its description and its rows, and the row's own configure control opens the same form under the plugin's title. It edits the web-fetch-playwright settings section — derived from the plugin entry's Config schema — live:

Field	Default	Description
backend	local	Radio: Local Playwright or Remote CDP endpoint, each with its own nested input.
playwrightPath	(blank)	Local backend: path to a playwright executable or a Chromium-family browser binary. Blank = discover on $PATH, then fall back to the bundled playwright-core.
cdpEndpoint	127.0.0.1:9222	Remote backend: host:port, http(s)://… or ws(s)://….
shareBrowserContext	true	CDP backend only. Checked (profile mode): each fetch is a tab in the remote browser's default context — its real profile — so cookies/localStorage are shared and its persistent logins apply; only the tab closes when the fetch ends. Unchecked (isolated mode): a fresh incognito-like context per fetch, nothing shared. Local backend ignores this field.
denoise	true	Run the denoise pipeline; off returns the full rendered HTML for the tool layer to convert.
maxConcurrency	(auto)	How many fetches may render at once (1–200). Blank = backend default: 4 for local (each slot launches a browser) / 50 for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within 20s they fail with WEB_FETCH_TIMEOUT and a hint to retry or raise this setting, rather than hanging until the tool budget aborts.
challengeWaitMs	15000	Bounded wait (ms, 0–60000) for a Cloudflare challenge to clear naturally in the same tab. 0 disables the whole challenge path — the first response is returned as-is (the pre-0.2.5 behavior).
challengeRetries	1	Same-tab re-navigation attempts after a wait window runs out (0–3); any clearance cookies the browser earned stay in the context for the retry. Everything stays inside the 45s per-fetch deadline.

Local backend resolution order:

• The configured path (auto-detected as Playwright CLI or browser binary).
• A playwright executable on $PATH (its package knows that installation's browser registry).
• The bundled playwright-core — requires PLAYWRIGHT_BROWSERS_PATH or browsers in the default cache; otherwise the error suggests playwright install chromium.

Windows note — $PATH is scanned with the platform delimiter (;), but npm/pnpm global installs expose playwright as .cmd/.ps1 shims whose location does not walk up to the package root, so discovery may still land on step 3 (the bundled core). To drive a specific installation's browser registry, point playwrightPath at the playwright package directory or a browser executable.

CDP mode needs no local browser: the provider holds one shared connection for its lifetime (reconnecting automatically if it drops, and reconnecting to the new endpoint when the setting changes), and every fetch leases a tab in the remote browser that closes when the fetch completes. Concurrency therefore counts tabs, which is why the CDP default is high (50). Unloading the plugin drops the shared connection (the remote browser itself is never closed).

The CDP context modes (share the browser's profile or not)

With Share the browser context checked (default, profile mode), each fetch is a tab in the remote browser's default context — the real profile. Cookies and localStorage come from and are written back to that profile, so sites the browser is logged into are fetched logged-in, exactly like a tab you open by hand. The shared context is never closed; resource filtering and popup guards attach to the fetch's tab only, so your other tabs are untouched. With it unchecked (isolated mode), every fetch gets a fresh incognito-like context — anonymous reads, nothing persists.

Profile-mode risk notes — it upgrades web_fetch from "anonymous read" to "acts as the browser's logged-in user":

• A fetched page that talks the agent into a GET-style state-changing URL (logout, settings change, API call) will send it with your session cookies.
• Concurrent fetches to the same site share one cookie jar; one fetch's logout or Set-Cookie affects the others.
• Output starts depending on the browser's history (A/B buckets, language preferences). The remote profile also keeps accumulating site data; the plugin never cleans it.

Persistent logins require a persistent user-data-dir. Headful (recommended — log in by hand once):

google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/chrome-dsh-profile"

Headless server (pre-seed logins in a headful environment first): chromium --headless=new --remote-debugging-port=9222 --user-data-dir=/data/chrome-dsh-profile. Do not add --incognito or a throwaway user-data-dir — either defeats profile mode. Design notes and verified playwright-core facts live in docs/context-mode-profile.md.

Cloudflare challenge handling (bounded natural wait)

Some strict sites serve a Cloudflare interstitial before the real page. A real browser often passes the check on its own within a few seconds — but a fetch that only looks at the first response hands you the interstitial as if it were the page (the pre-0.2.5 behavior; reproduce it any time with challengeWaitMs: 0, or — from a repo checkout, after pnpm build — run node scripts/challenge-demo.mjs for a local simulated before/after, and node scripts/challenge-online.mjs <url> against a real site).

With the wait on (default):

• Detection — a response carrying cf-mitigated: challenge (the documented signal for every challenge page type), or a 403/503 HTML document from a server: cloudflare edge, or the localized interstitial itself (title family like "Just a moment…", 请稍候…, "Минутку…", plus structural markers: the challenge-platform runtime's <script src>, #challenge-* elements, cf-chl-widget- frames, window._cf_chl_opt). The content-level markers are the fallback tier and only run on challenge-compatible responses — 403/429/503 or a Cloudflare edge (server: cloudflare / cf-ray) — because interstitials never ship as a plain 200, so an ordinary article that merely quotes challenge text can never be mistaken for one. Two refinements keep that promise for real pages: the challenge-platform script check reads real <script src> attributes only (prose, and escaped code samples that quote a path, are ignored), and it matches the runtime mount /cdn-cgi/challenge-platform/h/[b|g]/orchestrate/ positively. Everything else Cloudflare injects under that prefix — Bot Management's JavaScript Detections (scripts/jsd/, h/[b|g]/scripts/jsd/<hash>/, h/[b|g]/jsd/r/), and Precursor (scripts/precursor/) — is passive instrumentation present on every normal page of a protected zone (e.g. openrouter.ai, nodeseek.com), so it is never read as a challenge. A hard block ("Sorry, you have been blocked") is classified separately and fails immediately.
• Bounded wait, same tab and context — the fetch polls the live DOM (500ms interval) for the challenge to disappear while the browser runs its own verification; the last main-frame navigation response is tracked so the reloaded real document's status and headers are the ones reported. SPA-style clears (content swapped without any navigation) are caught by the same DOM probe.
• Bounded retry — when a window runs out, the same tab re-navigates once (default; challengeRetries) with whatever clearance cookies the context already holds.
• Clear failure — WEB_FETCH_CHALLENGE (a provider-specific code the web seam's open-string code allows) naming the site, the budget spent, and the last challenge status.

Security boundary (deliberate): no clicking through Turnstile, no CAPTCHA solving or token injection, no fingerprint/UA spoofing, no proxy rotation, and no cookie export — in isolated mode the clearance a fetch's browser earns dies with that fetch's context; in profile mode it stays in the remote browser's own profile, which this plugin never copies or cleans. The wait is always bounded by challengeWaitMs and the 45s per-fetch deadline; nothing blocks forever.

Development

pnpm install
pnpm typecheck   # tsc --noEmit
pnpm test        # vitest run (browser smoke self-skips without a browser)
pnpm build       # tsc declarations + tsdown (host ESM + client module-registration bundle)

Repository layout:

src/
├── index.ts               # host entry: registers the fetch provider (the settings section derives from the plugin's Config schema)
├── config.ts              # schemastery schema, CDP endpoint normalizer
├── provider.ts            # WebFetchProvider: navigation, deadline, semaphore, caps
├── markdown.ts            # denoise pipeline (Readability + DOMPurify + Turndown/GFM)
├── playwright-resolve.ts  # local backend discovery (path / $PATH / bundled core)
├── types.ts               # structural Playwright types (runtime module discovered dynamically)
└── client/                # browser half: settings card, form model, locales
tests/                     # unit + provider + browser integration (self-skipping)

See CONTRIBUTING.md for the development and release workflow, and SECURITY.md for the security model and reporting policy.

Security

Same stance as the built-in HTTP provider: no SSRF / private-network protection is implemented — anything the browser can reach, this provider can fetch. The CDP endpoint is configured from the settings page with no loopback restriction, so only expose the settings page to trusted environments. Fetched pages are rendered locally; no data is sent anywhere beyond the target page itself — but note that in profile mode the requests (and any state changes a malicious page tricks the agent into) carry the remote browser's logged-in sessions (see the risk notes above).

License

MIT © 2026 chendefine