@spicrawl/sdk

Spicrawl SDK for JavaScript & TypeScript

@spicrawl/sdk is the official TypeScript/JavaScript client for the Webora web scraping and web data API: it fetches any URL, renders JavaScript, converts pages to Markdown for LLMs, extracts structured data, and runs large batch crawls from Node.js.

• Zero runtime dependencies: native fetch and AbortController, Node.js 18+.
• Fully typed: request and response types are generated from the Webora OpenAPI spec.
• ESM and CommonJS, tree-shakeable.
• Billing-safe retries: a scrape is never retried in a way that could bill you twice.
• Pagination built in: for await over any list.

Contents

• Install
• Quickstart
• Scrape
• Batch
• Sessions
• Request history
• Usage
• Remote browser (coming soon)
• Response metadata
• Error handling
• Retries
• Timeouts and cancellation
• Pagination
• FAQ
• Related packages

Install

npm install @spicrawl/sdk

Get an API key from your Webora dashboard and export it:

export WEBORA_API_KEY=webora_live_...

Quickstart

import { Webora } from "@spicrawl/sdk";

const webora = new Webora(); // reads WEBORA_API_KEY

const page = await webora.scrape({ url: "https://example.com" });
console.log(page.status);  // the target site's HTTP status
console.log(page.content); // the page HTML

CommonJS works too:

const { Webora } = require("@spicrawl/sdk");

Client options

const webora = new Webora({
  apiKey: "webora_live_...",          // default: process.env.WEBORA_API_KEY
  baseURL: "https://api.webora.dev",  // default: process.env.WEBORA_BASE_URL, else https://api.webora.dev
  timeoutMs: 180_000,                 // per attempt
  maxRetries: 2,
  fetch: customFetch,                 // e.g. undici with a proxy agent
  defaultHeaders: { "X-Team": "growth" },
});

Scrape

scrape() calls POST /v1/scrape and returns the typed JSON envelope: content, the target's status and headers, data from extraction, links, screenshots, credits and engine.

// Plain HTTP fetch (cheapest)
const r = await webora.scrape({ url: "https://example.com" });

// Render JavaScript
const spa = await webora.scrape({ url: "https://example.com/app", js_render: true, wait_for: "#root" });

// Extract fields with CSS selectors
const product = await webora.scrape({
  url: "https://example.com/product/1",
  extract: { title: "h1", price: ".price" },
});
console.log(product.data, product.empty_fields);

// Let the platform choose the cheapest engine that works
const auto = await webora.scrape({ url: "https://example.com", mode: "auto" });

// Collect every link on the page
const { links } = await webora.scrape({ url: "https://example.com", links: true });

Raw formats: HTML, Markdown, text, PDF

Use scrapeRaw() when you want the document itself instead of the envelope. It returns a union discriminated by format:

const md = await webora.scrapeRaw({ url: "https://example.com", response_format: "markdown" });
if (md.format === "markdown") console.log(md.body); // string

const pdf = await webora.scrapeRaw({ url: "https://example.com", response_format: "pdf", engine: "chromium" });
if (pdf.format === "pdf") await fs.writeFile("page.pdf", Buffer.from(pdf.body)); // ArrayBuffer

format	body	Notes
html, markdown, text	string
pdf	ArrayBuffer	needs a browser engine (engine: "chromium")
json	string	also envelope; returned when you ask for JSON, or when extract, links, screenshot, network_capture, autoparse or ai_extract coerce the response

The target's status is on targetStatus (X-Target-Status), and credits, engine and warnings are on _meta.

The target's status is not an error

A site that answers 404 or 403 is still a successful API call. page.status (or targetStatus) is the site's answer, and the SDK only throws for platform failures. With original_status: true the API relays the target's status as the HTTP status. The SDK recognises this (the response carries X-Engine) and returns the content instead of throwing.

Batch

Queue up to 10,000 URLs as one asynchronous job.

const job = await webora.batch.create({
  urls: ["https://example.com/1", "https://example.com/2"],
  js_render: true,
});

const done = await webora.batch.waitForCompletion(job.id, {
  pollIntervalMs: 3000,
  timeoutMs: 30 * 60_000,
  onProgress: (j) => console.log(j.status, j.progress),
});

// Streams JSON Lines across every results page
for await (const line of webora.batch.results(job.id)) {
  if (line.status === "succeeded") console.log(line.url, line.http_status);
  else console.log(line.url, line.error?.code);
}

Other batch calls:

await webora.batch.get(id);
for await (const j of webora.batch.list({ status: "running" })) console.log(j.id);
await webora.batch.cancel(id);
await webora.batch.retry(id);                       // re-run failed items
await webora.batch.append(id, { items: [{ url }] }); // add to an open job
await webora.batch.close(id);                       // let an open job finish
const page = await webora.batch.resultsPage(id, { limit: 100, status: "failed" });
const { content, truncated } = await webora.batch.taskContent(id, 0);

Per-item overrides and correlation ids:

await webora.batch.create({
  items: [{ url: "https://example.com/a", external_id: "sku-1" }, { url: "https://example.com/b", js_render: false }],
  js_render: true,
});

Sessions

A session keeps cookies and storage across scrapes, so you can log in once and reuse the login.

const session = await webora.sessions.create({ engine: "obscura" });
await webora.scrape({ url: "https://example.com/account", session_id: session.id, js_render: true });

await webora.sessions.get(session.id);
for await (const s of webora.sessions.list()) console.log(s.id, s.status);
const ctx = await webora.sessions.context(session.id); // cookies + storage (contains credentials)
await webora.sessions.release(session.id);
await webora.sessions.delete(session.id, { force: true });

Request history

Every request is logged with a full trace, readable within your retention window. This needs the read scope.

for await (const entry of webora.requests.list({ only_errors: "true" })) {
  console.log(entry.id, entry.status);
}
const trace = await webora.requests.get(requestId);

Usage

const summary = await webora.usage.summary();
const byDay = await webora.usage.get({ from: "2026-09-01", to: "2026-09-24", group_by: "day" });
const recon = await webora.usage.reconciliation({ day: "2026-09-23" });

Remote browser (coming soon)

Remote CDP browsers (/v1/browser) and managed proxy pools are coming soon and not available during the beta. The SDK already includes webora.browser.token() and webora.browser.connectURL() so you won't need to change code when they launch. During the beta, pass your own proxy with proxy: "http://user:pass@host:port".

Response metadata

Every result carries a non-enumerable, typed _meta read from the response headers. It stays out of JSON.stringify and object spreads.

const r = await webora.scrape({ url: "https://example.com" });
r._meta.requestId;      // X-Request-Id: pass to webora.requests.get()
r._meta.creditsCharged; // X-Credits-Charged
r._meta.requestCost;    // X-Request-Cost
r._meta.engine;         // X-Engine
r._meta.cacheState;     // "hit" | "miss" | "bypass"
r._meta.targetStatus;   // X-Target-Status
r._meta.rateLimit;      // { limit, remaining, reset }
r._meta.warnings;       // X-Warning values, "CODE: message"
r._meta.headers;        // all raw headers

Pages from list methods have _meta too. scrapeRaw(), batch.resultsPage() and batch.taskContent() return it as a regular field.

Error handling

Every failure throws a WeboraError subclass:

Class	When
BadRequestError	400 (and other 4xx without a dedicated class)
AuthenticationError	401
InsufficientCreditsError	402
PermissionDeniedError	403
NotFoundError	404
ConflictError	409
GoneError	410
RateLimitError	429
InternalServerError	5xx
APIConnectionError	no response (DNS, refused, reset)
APITimeoutError	timeoutMs exceeded

import { isWeboraError, RateLimitError } from "@spicrawl/sdk";

try {
  await webora.scrape({ url: "https://example.com" });
} catch (err) {
  if (isWeboraError(err)) {
    err.code;              // "ERR::UPSTREAM::CHALLENGE": switch on this
    err.status;            // platform HTTP status
    err.retryable;         // the server's own verdict
    err.retryAfterSeconds;
    err.targetStatus;      // the site's status, when the site caused it
    err.diagnostics?.hint; // the parameter to change next
    err.requestId;         // for webora.requests.get()
    err.docUrl;
    err.problem;           // the raw RFC 7807 body
  }
}

ERROR_CODES lists every documented code, and the ErrorCode type autocompletes them.

Retries

The Webora API has no idempotency keys, so a retried scrape is a second, billed scrape. For that reason the SDK retries conservatively. The default is maxRetries: 2, with exponential backoff and jitter (about 0.5 s, then 1 s, capped at 8 s). A Retry-After header or retry_after_seconds value of up to 60 s is honoured.

	GET / DELETE	POST (scrape, batch create/append, sessions, …)
Connection error	retried	retried only when the connection never opened (ECONNREFUSED, DNS failure)
Timeout	retried	never, because the server may still be working and billing
408 / 5xx	retried unless the problem says retryable: false	never
429	retried	retried (rejected before any work was done)

Set a different value per client or per call:

const webora = new Webora({ maxRetries: 0 });
await webora.sessions.get(id, { maxRetries: 5 });

To retry a failed scrape yourself, check err.retryable first. Failed scrapes cost 0 credits.

Timeouts and cancellation

Every method takes a final options argument:

const controller = new AbortController();
await webora.scrape({ url }, { timeoutMs: 60_000, signal: controller.signal, headers: { "X-Trace": "abc" } });

timeoutMs applies to each attempt. The default is 180 s, because browser renders can be slow.

Pagination

List methods (batch.list, sessions.list, requests.list) return a promise-like page that is also async-iterable. The SDK handles both of the API's pagination styles, cursor (next_cursor) and keyset (before + before_id):

// Every item, across every page
for await (const s of webora.sessions.list({ limit: 100 })) console.log(s.id);

// One page at a time
let page = await webora.requests.list({ limit: 50 });
console.log(page.data, page.body.page.retention_hours);
while (page.hasNextPage()) page = await page.getNextPage();

FAQ

How do I scrape a JavaScript-rendered page in Node.js?

Pass js_render: true. Webora renders the page in a headless browser engine and returns the resulting DOM. Add wait_for: "<css selector>" to wait for content to appear:

const r = await webora.scrape({ url: "https://example.com/app", js_render: true, wait_for: ".loaded" });

How do I convert a web page to Markdown for an LLM?

Request response_format: "markdown" with scrapeRaw(). Add main_content_only: true to drop navigation and boilerplate:

const r = await webora.scrapeRaw({ url, response_format: "markdown", main_content_only: true });
if (r.format === "markdown") prompt += r.body;

How do I extract structured data (JSON) from a web page?

Pass extract with a map of field names to CSS selectors, or autoparse: true. The result is in data, and empty_fields lists the selectors that matched nothing:

const { data } = await webora.scrape({ url, extract: { title: "h1", price: ".price" } });

How do I scrape thousands of URLs?

Use webora.batch.create({ urls }), then waitForCompletion() and for await (const line of webora.batch.results(id)). A job holds up to 10,000 URLs per call and can be extended with append().

Why didn't my scrape throw when the site returned 404?

The SDK distinguishes two statuses. The HTTP status is the platform's, and the site's status is status in the envelope, targetStatus in raw mode, or _meta.targetStatus. A site that refused you is still a successful API call.

Will retries double-bill me?

No. POST requests are retried only on 429 or when the connection never opened. See Retries.

How do I use my own proxy?

Pass proxy: "http://user:pass@host:port" in the scrape params. Managed proxy pools are coming soon.

Does it work in Bun, Deno, Cloudflare Workers or the browser?

The SDK only uses the standard fetch, AbortController, Headers and Response APIs, so it runs in any runtime that provides them. Never ship your API key to a browser.

How do I debug a failed request?

Pass err.requestId (or result._meta.requestId) to webora.requests.get() to see the full trace, and check err.diagnostics.hint.

Related packages

• CLI: @spicrawl/cli (the spicrawl command) runs Webora from your terminal.
• MCP server: webora-mcp-server provides Webora as native tools for Claude, Cursor and other AI agents.
• API reference: https://api.webora.dev/docs

License

Apache-2.0