@spicrawl/sdk
Advanced tools
Official TypeScript SDK for the Spicrawl web scraping API: scrape any URL to LLM-ready Markdown or structured JSON, batch crawl and extract data.
Spicrawl is a web scraping API that turns any URL into clean Markdown, HTML or structured JSON for LLMs, RAG pipelines and AI agents. @spicrawl/sdk is the official TypeScript client for scraping pages, batch-crawling lists of URLs and extracting structured data, with full type definitions and zero dependencies.
Docs: https://docs.spicrawl.com · CLI: @spicrawl/cli · MCP server for AI agents: https://docs.spicrawl.com/agents/mcp
Spicrawl is a web data API: you send it a URL and it returns the page as HTML, Markdown, plain text, PDF or a JSON envelope with extracted fields. It can render JavaScript in a headless browser, extract data with CSS selectors, XPath, page metadata or AI, and process lists of up to 10,000 URLs as one batch job. This SDK wraps that HTTP API for Node.js 18+, with ESM and CommonJS builds and TypeScript types generated from the OpenAPI spec.
npm install @spicrawl/sdk
# or: pnpm add @spicrawl/sdk / yarn add @spicrawl/sdk / bun add @spicrawl/sdk
Requires Node.js 18 or later. Create an API key in your Spicrawl dashboard (keys start with webora_live_ or webora_test_) and export it:
export WEBORA_API_KEY=webora_live_...
import { Webora } from "@spicrawl/sdk";
const webora = new Webora(); // reads WEBORA_API_KEY
const page = await webora.scrape({ url: "https://example.com" });
console.log(page.status); // the target site's HTTP status, e.g. 200
console.log(page.content); // the page HTML
The client class is named Webora and talks to Spicrawl's API at https://api.webora.dev.
links: true, to feed your own crawl queue.scrape() returns a typed JSON envelope. scrapeRaw() returns the document itself, as a union discriminated by format. Use it to get LLM-ready Markdown:
const md = await webora.scrapeRaw({
url: "https://example.com/blog/post",
response_format: "markdown",
main_content_only: true, // strip nav, footer and sidebars
});
if (md.format === "markdown") console.log(md.body); // string
format | body type | Notes |
|---|---|---|
html, markdown, text | string | |
pdf | ArrayBuffer | needs a browser engine (engine: "chromium") |
json | string, plus envelope | returned when you ask for JSON, or when extract, links, screenshot, autoparse, ai_extract or network_capture is set |
Use include_tags and exclude_tags (CSS selectors) to keep or drop parts of the page before conversion.
Set js_render: true to load the page in a headless browser so client-side JavaScript runs. Add wait_for to wait for an element to appear:
const spa = await webora.scrape({
url: "https://example.com/app",
js_render: true,
wait_for: "#root .loaded",
});
Not sure whether a site needs a browser? mode: "auto" starts with a plain HTTP fetch and escalates only when needed, billing only the step that worked:
const auto = await webora.scrape({ url: "https://example.com", mode: "auto" });
console.log(auto._meta.engine); // the engine that produced the page
const product = await webora.scrape({
url: "https://example.com/product/1",
extract: { title: "h1", price: ".price" },
});
console.log(product.data); // { title: "...", price: "..." }
console.log(product.empty_fields); // selectors that matched nothing
const meta = await webora.scrape({ url: "https://example.com/article", autoparse: true });
console.log(meta.data);
Describe what you want in plain language, optionally with a JSON Schema for the output:
const ai = await webora.scrape({
url: "https://example.com/pricing",
ai_extract: {
prompt: "List each plan with its name and monthly price",
schema: { type: "object", properties: { plans: { type: "array" } } },
},
});
console.log(ai.data);
const { links } = await webora.scrape({ url: "https://example.com", links: true });
Screenshots and PDFs need a rasterising browser engine, so pin engine: "chromium". Images come back base64-encoded under screenshots:
import { writeFile } from "node:fs/promises";
const shot = await webora.scrape({
url: "https://example.com",
engine: "chromium",
screenshot: true,
screenshot_fullpage: true,
});
const img = shot.screenshots?.[0];
if (img) await writeFile("page.png", Buffer.from(img.data, "base64"));
const pdf = await webora.scrapeRaw({ url: "https://example.com", response_format: "pdf", engine: "chromium" });
if (pdf.format === "pdf") await writeFile("page.pdf", Buffer.from(pdf.body));
Use screenshot_selector to capture one element instead of the full page.
Queue up to 10,000 URLs as one asynchronous job, wait for it, then stream the results:
const job = await webora.batch.create({
urls: ["https://example.com/1", "https://example.com/2"],
js_render: true,
});
await webora.batch.waitForCompletion(job.id, {
pollIntervalMs: 3000,
timeoutMs: 30 * 60_000,
onProgress: (j) => console.log(j.status, j.progress),
});
// Streams JSON Lines across every results page
for await (const line of webora.batch.results(job.id)) {
if (line.status === "succeeded") console.log(line.url, line.http_status);
else console.log(line.url, line.error?.code);
}
Per-item overrides and your own correlation ids:
await webora.batch.create({
items: [
{ url: "https://example.com/a", external_id: "sku-1" },
{ url: "https://example.com/b", js_render: false },
],
js_render: true,
});
Other batch calls: get(id), list({ status }), cancel(id), retry(id) (re-run failed items), append(id, { items }) and close(id) for open jobs, resultsPage(id, { limit, status }) and taskContent(id, seq).
A session keeps cookies and web storage across scrapes, so a login carries over to later requests:
const session = await webora.sessions.create();
await webora.scrape({ url: "https://example.com/account", session_id: session.id, js_render: true });
const ctx = await webora.sessions.context(session.id); // cookies + storage: contains credentials, do not log
await webora.sessions.release(session.id);
await webora.sessions.delete(session.id, { force: true });
Pass a proxy URL (http, https, socks5 or socks5h, credentials in the userinfo). proxy_verify: true checks the proxy first so a dead one fails fast:
await webora.scrape({
url: "https://example.com",
proxy: "http://user:pass@proxy.example.net:8080",
proxy_verify: true,
});
Spicrawl is in beta. What that means today:
proxy. Managed proxy pools (premium_proxy, proxy_country) are coming soon.webora.browser.token() / connectURL()) is coming soon. The methods exist so your code won't change when it launches.links: true.List methods (batch.list, sessions.list, requests.list) return a page you can await or iterate with for await, which follows every following page automatically:
// Every item, across every page
for await (const s of webora.sessions.list({ limit: 100 })) console.log(s.id);
// One page at a time
let page = await webora.requests.list({ limit: 50 });
console.log(page.data.length);
while (page.hasNextPage()) page = await page.getNextPage();
A target site answering 404 or 403 is not an error: page.status holds the site's answer and the call succeeds. The SDK throws only for platform failures, always as a WeboraError subclass:
| Class | When |
|---|---|
BadRequestError | 400 (and other 4xx without a dedicated class) |
AuthenticationError | 401 |
InsufficientCreditsError | 402 |
PermissionDeniedError | 403 |
NotFoundError | 404 |
ConflictError | 409 |
GoneError | 410 |
RateLimitError | 429 |
InternalServerError | 5xx |
APIConnectionError | no response (DNS, refused, reset) |
APITimeoutError | timeoutMs exceeded |
import { isWeboraError, RateLimitError } from "@spicrawl/sdk";
try {
await webora.scrape({ url: "https://example.com" });
} catch (err) {
if (err instanceof RateLimitError) console.log("retry in", err.retryAfterSeconds, "s");
else if (isWeboraError(err)) {
console.log(err.code); // e.g. "ERR::UPSTREAM::TIMEOUT": switch on this
console.log(err.retryable); // the server's own verdict
console.log(err.diagnostics?.hint); // which parameter to change next
console.log(err.requestId); // pass to webora.requests.get()
}
}
ERROR_CODES lists every documented code, and the ErrorCode type autocompletes them.
The API has no idempotency keys, so a retried scrape is a second, billed scrape. The SDK therefore retries conservatively: maxRetries: 2 by default, exponential backoff with jitter, and Retry-After honoured up to 60 s.
| GET / DELETE | POST (scrape, batch create/append, sessions) | |
|---|---|---|
| Connection error | retried | retried only if the connection never opened |
| Timeout | retried | never (the server may still be working) |
| 408 / 5xx | retried unless retryable: false | never |
| 429 | retried | retried (rejected before any work) |
const strict = new Webora({ maxRetries: 0 });
await webora.sessions.get("sess_123", { maxRetries: 5 });
Failed scrapes cost 0 credits, and cache hits cost 0 credits.
Every method takes a final options argument with timeoutMs (per attempt, default 180 s), an AbortSignal and extra headers:
const controller = new AbortController();
await webora.scrape(
{ url: "https://example.com" },
{ timeoutMs: 60_000, signal: controller.signal, headers: { "X-Trace": "abc" } },
);
Every result carries a typed, non-enumerable _meta read from the response headers. It stays out of JSON.stringify and object spreads.
const r = await webora.scrape({ url: "https://example.com" });
r._meta.requestId; // pass to webora.requests.get() for the full trace
r._meta.creditsCharged; // what this call billed
r._meta.engine; // engine that produced the page
r._meta.cacheState; // "hit" | "miss" | "bypass"
r._meta.rateLimit; // { limit, remaining, reset }
r._meta.warnings; // non-fatal "CODE: message" warnings
for await (const entry of webora.requests.list({ only_errors: "true" })) console.log(entry.id);
const summary = await webora.usage.summary(); // plan, credits, current-period totals
const byDay = await webora.usage.get({ from: "2026-09-01", to: "2026-09-24", group_by: "day" });
const client = new Webora({
apiKey: "webora_live_...", // default: process.env.WEBORA_API_KEY
baseURL: "https://api.webora.dev", // default: process.env.WEBORA_BASE_URL
timeoutMs: 180_000, // per attempt
maxRetries: 2,
defaultHeaders: { "X-Team": "growth" },
// fetch: customFetch, // e.g. undici with an agent, or a mock in tests
});
Request and response types are generated from the OpenAPI spec and exported by name:
import type { ScrapeRequest, ScrapeEnvelope, BatchJob, ErrorCode } from "@spicrawl/sdk";
const { Webora } = require("@spicrawl/sdk");
Install @spicrawl/sdk, set WEBORA_API_KEY, and call await new Webora().scrape({ url }). The result holds the page content, the site's status and headers, and any extracted data.
Call webora.scrapeRaw({ url, response_format: "markdown" }) and read body when format === "markdown". The SDK needs no HTML parser or extra dependency; the conversion happens on Spicrawl's side.
Call scrapeRaw({ url, response_format: "markdown", main_content_only: true }) and use body when format === "markdown". Main-content isolation removes navigation, footers and sidebars so the Markdown is ready for a prompt or a RAG index.
Yes. Set js_render: true to render the page in a headless browser, and wait_for to wait for a selector; or use mode: "auto" to escalate from a plain fetch only when needed.
Pass extract with a map of field names to CSS selectors, autoparse: true for JSON-LD and OpenGraph metadata, or ai_extract with a prompt. The result is in data.
Use webora.batch.create({ urls }), then waitForCompletion() and for await (const line of webora.batch.results(id)). A job accepts up to 10,000 URLs and open jobs can grow with append().
Spicrawl is in beta. See https://docs.spicrawl.com for current plans and credit limits; failed scrapes and cache hits cost 0 credits.
Yes. Pass proxy: "http://user:pass@host:port" (HTTP, HTTPS or SOCKS5) on any scrape; it adds no proxy credits. Managed proxy pools are coming soon.
Retries never double-bill a scrape by accident. POST requests are retried only on 429 or when the connection never opened, both cases where no work was done. See Retries and billing safety.
The SDK is built on the standard fetch, AbortController, Headers and Response APIs and is supported on Node.js 18+. Runtimes that provide these APIs can run it, but never ship your API key to a browser: call Spicrawl from your server.
A Python SDK is in progress. Until it ships, you can call the HTTP API directly (see the docs) or use the spicrawl CLI.
The site's status is data, not an error: read it from status in the envelope, targetStatus in raw mode, or _meta.targetStatus. The SDK only throws when the platform itself fails.
@spicrawl/cli provides the spicrawl command for scraping from your terminal. CLI docsApache-2.0
FAQs
The Spicrawl TypeScript SDK (@spicrawl/sdk) is the official Node.js client for the Spicrawl web scraping API: it turns any URL into clean Markdown, HTML, JSON, a screenshot or a PDF, with JavaScript rendering, structured data extraction and batch jobs, fo
The npm package @spicrawl/sdk receives a total of 4 weekly downloads. As such, @spicrawl/sdk popularity was classified as not popular.
We found that @spicrawl/sdk demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Threat Research
Malicious Firefox extensions mimic Phantom, Jupiter, and Bitget wallets to steal recovery phrases and private keys, with one fetching its payload remotely.
Product
Socket Threat Intel now shows which malware campaigns were blocked or detected in your organization, with links to scans and Firewall events.
Security News
Fixes for eMudhra and Brotli are out. Getting them everywhere they’re needed means tracking down bundled libraries, runtimes, and certificate stores.