feedstock

Feedstock

High-performance web crawler and scraper for TypeScript, powered by Bun and Playwright.

Features

• Single & multi-page crawling with concurrent execution
• Deep crawling — BFS, DFS, and BestFirst traversal strategies
• Content extraction — CSS selectors, regex, XPath, table extraction
• Markdown generation with citation support
• Smart caching with ETag/Last-Modified validation via bun:sqlite
• URL filtering — pattern, domain, and content-type filters
• URL scoring — keyword relevance, path depth, freshness, domain authority
• Rate limiting — per-domain with exponential backoff
• Robots.txt parsing and compliance
• Built-in stealth mode — one flag enables random user-agents, navigator.webdriver override, plugin/language spoofing, human-like mouse/scroll simulation
• Anti-bot detection with auto-retry on blocked pages
• Multiple browser backends — Playwright (Chromium/Firefox/WebKit) or Lightpanda (local/cloud)
• Proxy rotation — round-robin strategy with health tracking
• URL seeding — discover URLs from sitemaps
• Accessibility snapshots — compact semantic page representation with @e refs for AI consumption
• Fetch-first engine system — tries lightweight HTTP before launching browser, auto-escalates for SPAs
• Rich metadata — 50+ fields: Open Graph, Twitter Cards, Dublin Core, JSON-LD, favicons, feeds
• Content processing — chunking (regex, sliding window, fixed-size) and filtering (pruning, BM25)
• Interactive element detection — finds all clickable elements including cursor:pointer and onclick handlers
• Storage state persistence — save/load cookies and localStorage between sessions
• AI-friendly errors — converts 20+ error patterns into actionable messages
• Hooks — inject custom behavior at 5 lifecycle points (page created, before/after navigation, etc.)
• Resource blocking — abort images, CSS, fonts, and media for faster content-only crawls
• Navigation strategies — configurable waitUntil: commit (fastest), domcontentloaded, load, networkidle
• In-page extraction — extract links/media/metadata directly in the browser via page.evaluate(), skipping HTML serialization
• Change tracking — detect new/changed/unchanged/removed pages between crawl runs with text diffs
• User-agent rotation — pool of 9 realistic browser user-agents with round-robin rotation
• Graceful shutdown — SIGINT/SIGTERM handlers auto-close browser processes
• Session management — LRU eviction at 20 concurrent sessions, cache TTL pruning
• Input validation — friendly error messages for invalid URLs, automatic retry on transient network errors
• Crawler monitoring — real-time stats tracking (pages/sec, success rates, data volume)
• Configurable logging — pluggable Logger interface with ConsoleLogger and SilentLogger

Quick Start

bun add feedstock
bunx playwright install chromium

import { WebCrawler, CacheMode } from "feedstock";

const crawler = new WebCrawler();

const result = await crawler.crawl("https://example.com", {
  cacheMode: CacheMode.Bypass,
});

console.log(result.markdown?.rawMarkdown);
console.log(result.links.internal);
console.log(result.media.images);

await crawler.close();

Deep Crawling

Recursively crawl entire sites with filters, rate limiting, and robots.txt compliance:

import {
  WebCrawler, CacheMode,
  FilterChain, DomainFilter, ContentTypeFilter, URLPatternFilter,
  RateLimiter, RobotsParser,
  CompositeScorer, KeywordRelevanceScorer, PathDepthScorer,
} from "feedstock";

const crawler = new WebCrawler();

const results = await crawler.deepCrawl(
  "https://example.com",
  { cacheMode: CacheMode.Bypass },
  {
    maxDepth: 3,
    maxPages: 100,
    concurrency: 5,
    filterChain: new FilterChain()
      .add(new DomainFilter({ allowed: ["example.com"] }))
      .add(new ContentTypeFilter())
      .add(new URLPatternFilter({ exclude: [/\/admin/, /\/login/] })),
    rateLimiter: new RateLimiter({ baseDelay: 500 }),
    robotsParser: new RobotsParser(),
    scorer: new CompositeScorer()
      .add(new KeywordRelevanceScorer(["docs", "api"], 2.0))
      .add(new PathDepthScorer()),
  },
);

Stream results as they arrive for large crawls:

for await (const result of crawler.deepCrawlStream(startUrl, {}, config)) {
  console.log(`Crawled: ${result.url}`);
}

Structured Extraction

CSS Selectors

const result = await crawler.crawl("https://example.com/products", {
  extractionStrategy: {
    type: "css",
    params: {
      name: "products",
      baseSelector: ".product",
      fields: [
        { name: "title", selector: "h2", type: "text" },
        { name: "price", selector: ".price", type: "text" },
        { name: "image", selector: "img", type: "attribute", attribute: "src" },
        { name: "tags", selector: ".tag", type: "list" },
      ],
    },
  },
});

const products = JSON.parse(result.extractedContent!)
  .map((item) => JSON.parse(item.content));

Tables

import { TableExtractionStrategy } from "feedstock";

const strategy = new TableExtractionStrategy({ minRows: 2 });
const tables = await strategy.extract(url, html);
// [{ headers: ["Name", "Age"], rows: [["Alice", "30"], ...], caption: "Users" }]

Regex

import { RegexExtractionStrategy } from "feedstock";

const strategy = new RegexExtractionStrategy([
  /\$(?<dollars>\d+)\.(?<cents>\d{2})/g,
]);
const prices = await strategy.extract(url, html);
// prices[0].metadata.groups = { dollars: "9", cents: "99" }

Browser Backends

Playwright (Default)

const crawler = new WebCrawler({
  config: {
    browserType: "chromium", // or "firefox", "webkit"
    headless: true,
  },
});

Lightpanda

// Local (requires: bun add @lightpanda/browser)
const crawler = new WebCrawler({
  config: {
    backend: { kind: "lightpanda", mode: "local" },
  },
});

// Cloud
const crawler = new WebCrawler({
  config: {
    backend: {
      kind: "lightpanda",
      mode: "cloud",
      token: process.env.LIGHTPANDA_TOKEN!,
    },
  },
});

Stealth Mode

One flag — random user-agents, navigator overrides, and human simulation:

// Enable stealth at browser level
const crawler = new WebCrawler({
  config: { stealth: true },
});

// Human simulation per-crawl
const result = await crawler.crawl(url, {
  simulateUser: true, // random mouse movements + scrolling
});

// Auto-retry on blocks
import { withRetry, isBlocked } from "feedstock";

const { result, retries } = await withRetry(
  () => crawler.crawl(url),
  (r) => isBlocked(r.html, r.statusCode ?? 200),
  { maxRetries: 3, retryDelay: 2000 },
);

Proxy Rotation

import { ProxyRotationStrategy } from "feedstock";

const rotation = new ProxyRotationStrategy([
  { server: "http://proxy1:8080" },
  { server: "http://proxy2:8080" },
  { server: "http://proxy3:8080" },
]);

const proxy = rotation.getProxy(); // round-robin, skips unhealthy
rotation.reportResult(proxy, true); // track health

Content Processing

Chunking

import { SlidingWindowChunking, RegexChunking } from "feedstock";

// Split by paragraphs
new RegexChunking().chunk(text);

// Sliding window with overlap
new SlidingWindowChunking(500, 50).chunk(text);

Content Filtering

import { PruningContentFilter, BM25ContentFilter } from "feedstock";

// Remove boilerplate
new PruningContentFilter({ minWords: 5 }).filter(content);

// Keep only relevant blocks
new BM25ContentFilter({ threshold: 0.1 }).filter(content, "TypeScript crawler");

Resource Blocking & Fast Navigation

// Block images/CSS/fonts for content-only crawl
const result = await crawler.crawl("https://example.com", {
  blockResources: true,
  navigationWaitUntil: "commit", // fastest — returns as soon as server responds
});

In-Page Extraction

Extract data directly in the browser — skips HTML serialization round-trip:

import { extractInPage } from "feedstock";

crawler.setHook("beforeReturnHtml", async (page) => {
  const data = await extractInPage(page);
  // data.links, data.media, data.metadata — extracted inside browser context
});

URL Discovery

import { URLSeeder } from "feedstock";

const seeder = new URLSeeder();
const { urls, sitemaps } = await seeder.seed("example.com");
// Discovers URLs from robots.txt -> sitemap.xml chain

Monitoring

import { CrawlerMonitor } from "feedstock";

const monitor = new CrawlerMonitor();
monitor.start();

// Track each page
monitor.recordPageComplete({
  success: true,
  fromCache: false,
  responseTimeMs: 150,
  bytesDownloaded: 45_000,
});

console.log(monitor.formatStats());
// Pages: 1 (1 ok, 0 failed, 0 cached)
// Time: 0.2s | 6.7 pages/s | avg 150ms/page
// Downloaded: 0.04 MB

Accessibility Snapshots

Compact semantic page representation — 3-10x smaller than HTML:

const result = await crawler.crawl("https://example.com", {
  snapshot: true,
});

console.log(result.snapshot);
// @e1 [heading] "Example Domain" [level=1]
// @e2 [link] "More information..." [-> https://www.iana.org/domains/example]

Hooks

Inject custom behavior at key lifecycle points:

crawler.setHook("afterGoto", async (page) => {
  // Dismiss cookie banners, expand sections, etc.
  const banner = page.locator('[class*="cookie"]');
  if (await banner.isVisible()) await banner.locator("button").first().click();
});

Available hooks: onPageCreated, beforeGoto, afterGoto, onExecutionStarted, beforeReturnHtml.

Interactive Element Detection

Find all clickable elements including those without ARIA roles:

import { detectInteractiveElements } from "feedstock";

crawler.setHook("beforeReturnHtml", async (page) => {
  const elements = await detectInteractiveElements(page);
  console.log(`Found ${elements.length} interactive elements`);
  // Each has: tag, text, href, role, type, selector
});

Storage State

Persist cookies/localStorage between sessions:

import { saveStorageState, loadStorageState } from "feedstock";

// Save after login
await saveStorageState(page.context());

// Load in next session
const state = loadStorageState();

Process HTML Without Browser

const html = "<html><body><h1>Hello</h1><p>World</p></body></html>";
const result = await crawler.processHtml(html, { snapshot: true });
console.log(result.markdown?.rawMarkdown);
console.log(result.snapshot);

Documentation

Full documentation at usefeedstock.com.

Development

bun install
bun test              # 277 tests
bun test tests/unit   # unit tests only
bun run typecheck     # tsc --noEmit
bun run lint          # biome check
bun run check         # lint + typecheck
bun run dogfood.ts    # 148 checks against real sites

License

Apache-2.0