🎩 You're Invited:Meet the Socket team at Black Hat in Las Vegas, August 3-6.RSVP
Sign In

imperium-crawl

Package Overview
Dependencies
Maintainers
1
Versions
26
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

imperium-crawl

41-tool open-source CLI for web scraping, PDF extraction, content monitoring, reusable browser flows, RSS aggregation, and custom skills. CamoFox C++ anti-detection engine. Zero API keys for core tools.

latest
npmnpm
Version
2.6.1
Version published
Weekly downloads
12
-86.36%
Maintainers
1
Weekly downloads
 
Created
Source
imperium-crawl — 4-level auto-escalating stealth engine with CamoFox

imperium-crawl

The most powerful open-source CLI tool for web scraping, crawling, and data extraction.

41 tools. CamoFox C++ anti-detection. Zero API keys required. One npx command.

npm version License: MIT Tests npm downloads

What's new in 2.6.0

CamoFox browser engine — Firefox fork with C++ anti-fingerprinting that bypasses Cloudflare, Google, and most bot detection:

  • C++-level patchesnavigator.hardwareConcurrency, WebGL, AudioContext, WebRTC spoofed before JavaScript sees them
  • Engine abstraction — Switch between Playwright and CamoFox with a single engine flag
  • Auto-updateimperiumcrawl camofox-update pulls the latest CamoFox release from npm
  • Engine factoryimport { resolveEngine } from "imperium-crawl/engines" for agent-native use
  • Zero breaking changes — Same tool API, same response format, same env vars. Just add engine: "camofox".
  • Codebase reorganized — CLI in src/cli/, core in src/core/, tests in 14 category folders.
  • 41 tools total — Added camofox_status and camofox_update.
# Download ALL images from any page (100% coverage)
imperium-crawl download <url> --images --output ./slike

# Target exactly the 3rd image
imperium-crawl download <url> --images --index 3

# Auto-click "Prikaži više" + scan iframes
imperium-crawl download <url> --images --auto-click --iframe-scan

See CHANGELOG.md for the full release notes.

Quick Start

Get running in 30 seconds.

CLI (zero install):

npx -y imperium-crawl scrape --url https://example.com

Global install:

npm install -g imperium-crawl

Install from a local tarball (e.g. pre-release testing):

npm install -g ./imperium-crawl-2.5.2.tgz

That's it. 33 of 41 tools work with zero API keys. Add optional keys later to unlock search, AI extraction, and CAPTCHA solving.

Power Examples

Real results. Copy-paste and try.

Scrape through Cloudflare

imperium-crawl scrape --url https://blog.cloudflare.com
Level 1 (headers) → blocked
Level 2 (TLS fingerprint) → blocked
Level 3 (browser + stealth) → success ✅
→ Full markdown content extracted, 213K characters
→ Next visit: skips straight to Level 3 (learned)

Discover hidden APIs on any website

imperium-crawl discover-apis --url https://weather.com
Found 11 hidden API endpoints:
  • api.weather.com — main weather API (exposed API key!)
  • mParticle analytics endpoints
  • Taboola content recommendation API
  • OneTrust consent management API
  • DAA/AdChoices opt-out endpoints
→ Call any endpoint directly with query_api — 10x faster than DOM scraping

AI extraction in plain English

imperium-crawl ai-extract --url https://amazon.com/dp/B0D1XD1ZV3 \
  --schema "extract product name, price, rating, and review count"
{
  "product_name": "Apple AirPods Pro 2",
  "price": "$189.99",
  "rating": "4.7 out of 5",
  "review_count": "45,297"
}

Extract ALL images from any page (100% coverage)

imperium-crawl download https://www.njuskalo.hr/nekretnine/stan-Zagreb --images --output ./slike
Discovered 23 unique images
  ✅ njuskalo.hr-001.jpg — 142KB
  ✅ njuskalo.hr-002.jpg — 89KB
  ✅ njuskalo.hr-003.jpg — 256KB
→ 23/23 downloaded. Total: 4.2MB

Target a specific image:

imperium-crawl download https://olx.ba/artikal/12345 \
  --images --selector "img.gallery-main" --output ./oglas.jpg

Auto-click "Load more" + iframe scan:

imperium-crawl download https://www.leboncoin.fr/ad/12345 \
  --images --auto-click --iframe-scan --limit 50

Batch scrape with resume

imperium-crawl batch-scrape \
  --urls '["https://bbc.com","https://cnn.com","https://reuters.com","https://techcrunch.com"]' \
  --concurrency 3
Scraping 4 URLs (concurrency: 3)...
  ✅ bbc.com — 47K chars
  ✅ cnn.com — 52K chars
  ✅ reuters.com — 38K chars
  ✅ techcrunch.com — 61K chars
→ 4/4 succeeded. Job ID: abc123 (resume with --job-id if interrupted)

Why imperium-crawl?

🔓 Zero API Keys Required 33 of 41 tools work out of the box. No accounts, no tokens, no credit cards. Just npx and go.

🛡️ 3-Level Auto-Escalating Stealth Headers → TLS fingerprinting → headless browser + CAPTCHA solving. Automatically escalates until it gets through.

🧠 Self-Improving Adaptive learning engine remembers what works per domain. Second visit is 3x faster. The more you use it, the smarter it gets.

🧰 41 Tools, 2 Modes CLI tool or interactive TUI. Scraping, crawling, search, extraction, API discovery, WebSocket monitoring, browser automation, batch processing.

📜 14 Built-in Recipes Pre-built workflows for common tasks — news extraction, e-commerce scraping, API reverse engineering, and more.

Skills System Teach it once, run forever. Auto-detect patterns on any page, save as reusable skills, get fresh data on demand.

vs. The Competition

Featureimperium-crawlFirecrawlCrawl4AIBrowserbasePuppeteer
PriceFree forever$19+/monthFree$0.01/minFree
Total tools41524N/A
Stealth levels3 (auto-escalate)Cloud-based1Cloud-basedNone
Anti-bot detection7 systemsPartialPartialPartialNone
TLS fingerprintingJA3/JA4NoNoNoNo
CAPTCHA auto-solvingYesNoNoNoNo
API discoveryYesNoNoNoNo
WebSocket monitoringYesNoNoNoNo
AI-powered extractionYesNoNoNoNo
Adaptive learningYesNoNoNoNo
Batch processingYesNoNoNoNo
ARIA SnapshotsYesNoNoNoNo
Session EncryptionYesNoNoNoNo
Self-hostedYesNoYesNoYes
Requires external serviceNoYesNoYesNo

Stealth Engine

Request → [L1: Headers + UA rotation]
              │
              ├─ success → Done
              ↓ fail
          [L2: TLS Fingerprint (JA3/JA4)]
              │
              ├─ success → Done
              ↓ fail
          [L3: Browser + Fingerprint Injection + CAPTCHA]
              │
              ├─ success → Done
              ↓
          [Learning Engine records optimal level for next time]

Stealth Levels

LevelMethodWhat It Defeats
1header-generator — Bayesian realistic headers + UA rotationBasic bot detection, simple WAFs
2impit — browser-identical TLS fingerprints (JA3/JA4)Cloudflare, Akamai, TLS fingerprinting WAFs
3rebrowser-playwright + fingerprint-injector + auto CAPTCHAJavaScript challenges, SPAs, advanced anti-bot, CAPTCHAs

Anti-Bot System Detection

Automatically identifies which anti-bot system a site uses and chooses the optimal strategy:

SystemDetection Method
Cloudflarecf_clearance cookies, cf-mitigated header, challenge page title
Akamai_abck, bm_sz cookies
PerimeterX / HUMAN_px cookies, _pxhd headers
DataDomedatadome cookies, datadome response header
Kasadax-kpsdk-* headers
AWS WAFaws-waf-token cookie
F5 / Shape SecurityTS prefix cookies

Smart Rendering Cache

Once imperium-crawl determines a domain needs Level 3 (browser), it caches that decision for 1 hour. Subsequent requests to the same domain skip straight to browser rendering — no wasted time on failed lower levels.

Adaptive Learning Engine

imperium-crawl learns from every request and gets smarter over time. No configuration needed — fully automatic.

Every time you scrape a website, the engine records which stealth level worked, which anti-bot system was detected, whether a proxy was needed, response timing, and success/failure. Next time you hit the same domain, it predicts the optimal configuration — skipping failed levels and going straight to what works.

First visit to cloudflare.com:
  Level 1 → blocked ❌
  Level 2 → blocked ❌
  Level 3 → success ✅ (Cloudflare detected)
  → Engine records: cloudflare.com needs Level 3

Second visit to cloudflare.com:
  → Engine predicts: Level 3, confidence 85%, Cloudflare
  → Skips Level 1 and 2 entirely — goes straight to browser
  → 3x faster than first visit

Smart Features

  • Time decay — Knowledge older than 7 days loses weight, adapts when sites change defenses
  • Confidence scoring — Low data = start from level 1. High confidence = skip to optimal level
  • Auto-prune — Domains unused for 30 days are cleaned up. Max 2,000 domains stored
  • Atomic persistence — Knowledge saved via atomic write (tmp → rename). Never corrupts

The more you use it, the faster it gets.

All 41 Tools

📄 Scraping (no API key needed)

ToolWhat It Does
scrapeURL to clean Markdown/HTML with 3-level auto-escalating stealth. Structured data (JSON-LD, OpenGraph, Microdata), metadata, and links.
crawlPriority-based crawling with depth control, concurrency limiting, and smart URL scoring.
mapDiscover all URLs on a domain via sitemap.xml + page link extraction.
extractCSS selectors to structured JSON. Point at any repeating pattern and get clean data.
readabilityMozilla Readability article extraction — title, author, content, publish date.
screenshotFull-page or viewport PNG screenshots via headless Chromium.

🔍 Search (requires free Brave API key)

ToolWhat It Does
searchWeb search via Brave Search API.
news_searchNews-specific search with freshness ranking.
image_searchImage search with thumbnails and source URLs.
video_searchVideo search across platforms.

⚡ Skills (no API key needed)

ToolWhat It Does
create_skillAnalyze any page, auto-detect repeating patterns, generate CSS selectors, save as reusable skill.
run_skillRun a saved skill for fresh structured data. Supports pagination.
list_skillsList all saved skills with configurations.

🔓 API Discovery & Real-Time (no API key needed, requires Playwright)

ToolWhat It Does
discover_apisNavigate to any page, intercept XHR/fetch calls, map hidden REST/GraphQL endpoints. Auto-detects GraphQL, filters noise, returns response previews.
query_apiCall any API endpoint directly with stealth headers. Bypass DOM rendering for 10x faster data access.
monitor_websocketCapture real-time WebSocket messages — financial tickers, chat feeds, live dashboards.

🧠 AI Extraction (requires LLM API key)

ToolWhat It Does
ai_extractDescribe what you want in natural language or JSON schema. 3 providers (Anthropic, OpenAI, MiniMax). The extract tool also supports llm_fallback: true for hybrid CSS→AI extraction.

🖱️ Interaction (no API key needed, requires Playwright)

ToolWhat It Does
interactBrowser automation with 20 action types (click, type, scroll, wait, screenshot, evaluate, select, hover, press, navigate, drag, upload, storage, cookies, pdf, auth_login, refresh, auto_click). Ref targeting via ARIA snapshot, session encryption, action policy, domain filter, network interception, device emulation. auto_click finds and clicks "load more" / "gallery" buttons with multilingual keyword matching.
snapshotARIA-based page snapshot with interactive element refs. Use refs in interact for precise targeting. Annotated screenshots.

📱 Social Media (no API key needed)

ToolWhat It Does
youtubeSearch videos, get video details, comments, transcripts, chapters, and channel info. Parses ytInitialData — no API key needed. Add OPENAI_API_KEY to unlock Whisper AI transcription for videos without captions.
redditSearch Reddit, browse subreddits, get posts and comments via Reddit's public JSON API.
instagramSearch profiles, get detailed profile info with engagement metrics, and discover influencers by niche/location. Search/discover require BRAVE_API_KEY.

📥 Media & Feeds (no API key needed)

ToolWhat It Does
downloadDownload media files from any URL — images, video, YouTube, TikTok, bulk. v2.5.1: Browser-based image extraction with 100% coverage (lazy-load, shadow DOM, iframes, JSON-LD, CSS backgrounds). Target specific images via --selector, --index, --alt-match. Auto-click "load more" buttons. Referer injection fixes 403 on CDNs.
batch_downloadDownload multiple files (PDFs, images, documents) in parallel with session cookie support. Uses L1 HTTP fetch — 10x faster than browser-based downloads. Ideal for bulk file retrieval from authenticated sessions.
rssFetch and parse RSS/Atom feeds. Filter by date, output as JSON or Markdown.

📦 Batch Processing (no API key needed)

ToolWhat It Does
batch_scrapeParallel URL scraping with configurable concurrency, soft failure, and resume via job_id. Optional AI extraction per URL.
list_jobsList all batch jobs with status and progress.
job_statusFull results for a specific batch job including per-URL outcomes.
delete_jobClean up completed or failed batch jobs.

🧠 Knowledge Engine (no API key needed)

ToolWhat It Does
knowledgeDump adaptive knowledge engine stats — per-domain success rates, optimal stealth levels, anti-bot detection history, rate limits. Use to debug scraping issues and understand problematic domains.

📄 Documents (no API key needed)

ToolWhat It Does
pdf_extractExtract text, pages, tables, and metadata from a local or remote PDF. Native text-layer strategy via pdfjs-dist. OCR + Claude Vision fallbacks deferred to v2.6.0. Use for sustainability reports, invoices, regulatory PDFs.
imperium-crawl pdf-extract --input ./report.pdf --output ./extracted.json
imperium-crawl pdf-extract --input https://example.com/report.pdf --max-pages 20

👀 Change Tracking (no API key needed)

ToolWhat It Does
watchOne-shot change detector: scrape a URL, hash its content (readability / markdown / full), compare against the last snapshot, fire a webhook on change. Pair with cron for periodic monitoring.
monitorPortfolio-level change tracker across many URLs grouped by topic. Reads a JSON config, runs watch on each URL, emits a markdown digest filtered by minimum change percentage.
# Watch a single URL — run periodically via cron
imperium-crawl watch --url https://carbonchain.com/pricing \
  --output-dir ./data/watch \
  --webhook https://hooks.example.com/on-change

# Monitor many URLs grouped by topic, emit a daily digest
imperium-crawl monitor --config ./monitor.json --output-dir ./data/monitor

monitor.json:

{
  "topics": [
    {
      "name": "Competitor pricing",
      "urls": ["https://carbonchain.com/pricing", "https://spherasolutions.com/cbam"]
    }
  ]
}

🔁 Imperium Flows (no API key needed; browser workflows may require Playwright)

ToolWhat It Does
record_flowRecord a headed browser workflow as a generic flow family/variant. Stores smart selector metadata and reusable input placeholders.
run_flowRun a saved flow with runtime JSON input, CAPTCHA policy, browser mode, and evidence collection.
serve_flowExpose saved flows through a local HTTP API. Requires bearer auth when bound publicly.
list_flowsList project-local and global flow definitions.
inspect_flowInspect a saved flow JSON definition.
validate_flowValidate a flow schema and report inputs, steps, and storage path.
imperium-crawl record-flow --family generic-search --variant site-a --url https://example.com
imperium-crawl run-flow generic-search/site-a --input '{"query":"example"}'
imperium-crawl serve-flow generic-search --port 8787

Setup

API Keys

KeyWhat It UnlocksWhere to Get It
BRAVE_API_KEY4 search tools (web, news, image, video)brave.com/search/api (free tier available)
TWOCAPTCHA_API_KEYAuto CAPTCHA solving (reCAPTCHA v2/v3, hCaptcha, Turnstile)2captcha.com
LLM_API_KEYAI-powered data extraction (ai_extract tool)Anthropic, OpenAI, or MiniMax API key
OPENAI_API_KEYWhisper AI transcription — transcribe any YouTube video, even without captionsplatform.openai.com
CHROME_PROFILE_PATHAuthenticated browser sessions (use your Chrome cookies)Path to Chrome user data dir
PROXY_URLRoute all requests through a proxy (http/https/socks4/socks5)Any proxy provider

Enable Full Stealth (Level 3)

npm i rebrowser-playwright
npx playwright install chromium

CLI Usage

With subcommand = runs that tool. No args in TTY = interactive TUI. No args in pipe = shows help.

# Scrape a website to markdown
imperium-crawl scrape --url https://bbc.com/news

# Crawl with depth control
imperium-crawl crawl --url https://blog.cloudflare.com --max-depth 2 --max-pages 5

# AI-powered extraction — plain English
imperium-crawl ai-extract --url https://amazon.com/dp/B0D1XD1ZV3 \
  --schema "extract product name, price, rating, and review count"

# Discover hidden APIs
imperium-crawl discover-apis --url https://weather.com

# Batch scrape in parallel
imperium-crawl batch-scrape --urls '["https://site1.com","https://site2.com"]' --concurrency 3

# Interactive setup wizard
imperium-crawl setup

Output Formats

imperium-crawl scrape --url https://example.com                          # JSON (default)
imperium-crawl scrape --url https://example.com --output-format markdown  # Markdown
imperium-crawl scrape --url https://example.com --output-format csv       # CSV
imperium-crawl scrape --url https://example.com --pretty                  # Pretty JSON
imperium-crawl scrape --url https://example.com --output result.json      # Write to file

TUI Mode

imperium-crawl tui

Interactive slash-command terminal with parameter prompts, table rendering, markdown display, and session state. Use /save to export results and /again to re-run the last command.

Explore REPL

Interactively explore a site in a headed browser, then save the session as a reusable skill:

imperium-crawl explore https://example.com
> navigate https://example.com/login
> type "#email" "user@example.com"
> type "#password" "{{env:MY_PASSWORD}}"
> click "#submit"
> snapshot
> save-skill my-login
✅ Saved skill: my-login (4 actions, 1 parameter detected)

Commands: navigate, click, type, select, hover, press, scroll, wait, screenshot, snapshot, evaluate, save-skill, history, undo, status, help, exit

Skills & Recipes

Skills let you teach imperium-crawl how to extract data from any website, then re-run for fresh content whenever you want.

Create a skill:

create_skill({
  url: "https://techcrunch.com/category/artificial-intelligence",
  name: "tc-ai-news",
  description: "Latest AI news from TechCrunch"
})

Run a skill:

run_skill({ name: "tc-ai-news" })
→ Returns fresh structured data with all detected fields

Skills are saved in ~/.imperium-crawl/skills/ as JSON files — human-readable, editable, portable.

Skill Parameters

Use template variables in skills — resolved at run time:

# In skill JSON actions:
{ "value": "{{input:query}}" }           # passed via --params or prompted
{ "value": "{{env:SITE_PASSWORD}}" }     # from environment variable
{ "value": "{{computed:date_today}}" }   # auto-computed (date_today, timestamp, random_string, year, month, day)

# Run with params:
imperium-crawl run-skill my-search --params '{"query": "machine learning"}'

Skill Chains

Chain skills together — output of one step becomes input to the next:

{
  "type": "chain",
  "name": "search-and-extract",
  "steps": [
    { "skill": "search-results", "output": "search" },
    { "skill": "extract-details", "input": { "url": "$search.results[0].url" }, "output": "details" }
  ]
}

Variable syntax: $step_name.field.nested[0] — simple dot-path access, no eval.

Built-in Recipes

RecipeWhat It Does
hn-top-storiesHacker News front page — titles, scores, comment counts
github-trendingGitHub trending repos — stars, language, description
job-listings-greenhouseGreenhouse job boards — title, team, location
ecommerce-productProduct name, price, rating, reviews, images
product-reviewsReview text, ratings, author, date from product pages
crypto-websocketLive crypto prices via WebSocket monitoring
news-article-readerArticle title, author, date, content from news sites
reddit-postsSubreddit posts — title, score, comments, flair
seo-page-auditSEO signals — meta tags, headings, structured data
social-media-mentionsBrand mentions across social platforms
influencer-niche-discoveryFind influencers by niche + location via Instagram
influencer-hashtag-scoutDiscover influencers through hashtag analysis
influencer-competitor-spyFind influencers from competitor brand mentions
influencer-content-scoutAnalyze content patterns of niche influencers

See SKILL/ for detailed workflow guides and agent integration.

API Discovery Workflow

Turn any website into an API. No documentation needed.

1. discover_apis({ url: "https://weather.com" })
   → Found 11 hidden API endpoints:
     • Main weather API (api.weather.com) with exposed API key
     • mParticle analytics endpoints
     • Taboola content recommendation API
     • OneTrust consent management API

2. query_api({ url: "https://api.weather.com/v3/...", method: "GET" })
   → Direct API call, bypasses DOM entirely — 10x faster, structured JSON

3. monitor_websocket({ url: "https://binance.com/en/trade/BTC_USDT", duration_seconds: 10 })
   → Captures real-time WebSocket messages — live BTC price feed

AI Agent Guide

imperium-crawl ships with SKILL/ — a structured guide that teaches AI agents how to use all 41 tools effectively. Includes proven workflows, decision trees, error recovery, and advanced patterns.

Two Ways to Connect

MethodSetupWorks With
CLI + SKILL/npm i -g imperium-crawl + SKILL.md in agent contextAny agent with bash access — Claude Code, Cursor, OpenClaw, ChatGPT, custom agents
TUIimperium-crawl tui — interactive terminalDirect human use, demos, debugging

Per-Agent Setup

AI AgentHow to Add SKILL/
Claude CodeCopy SKILL.md to project root — auto-detected
Cursor / WindsurfAdd SKILL.md to project rules or system prompt
OpenClaw / custom agentsInclude SKILL.md in system prompt or context window
ChatGPT / GPT agentsPaste SKILL.md content into custom instructions

Resilience

  • Exponential backoff with full jitter — AWS-recommended retry pattern, no thundering herd
  • Per-domain circuit breaker — 5 failures opens circuit for 60s, then half-open probing with auto recovery
  • URL normalization — 11-step pipeline removes tracking params (utm_*, fbclid, gclid), sorts query params
  • Proxy support — single proxy or rotating pool with http/https/socks4/socks5
  • Browser pool — keyed by proxy URL, auto-eviction, configurable pool size
  • robots.txt — respected by default (configurable)
  • Graceful shutdown — 10s timeout on browser cleanup to prevent hung processes

Real-World Test Results

Every tool tested against production websites with real anti-bot defenses:

ToolTargetResult
📄 scrapeBBC NewsFull markdown, stealth level 3 auto-escalation
🕸️ crawlCloudflare Blog213K characters crawled with depth control
🗺️ mapBBCFull URL discovery via sitemap + link extraction
🕷️ extractAmazon (AirPods Pro 2)Product title, 45,297 reviews, brand extracted
📖 readabilityMedium articleClean — title, author, content, publish date
📸 screenshotProductHuntCaptured Cloudflare Turnstile challenge page
🔍 searchBrave WebWeb results with snippets and URLs
📰 news_searchBrave NewsNews results with freshness ranking
🖼️ image_searchBrave ImageImages with thumbnails and source URLs
🎬 video_searchBrave VideoVideo results across platforms
🛠️ create_skillHacker NewsAuto-detected 30 stories with CSS selectors
▶️ run_skillSaved skillFresh structured data from saved config
📋 list_skillsLists all skills with configurations
🔓 discover_apisAirbnb Paris34 hidden APIs — DataDome, Google Maps key, internal APIs
query_apijsonplaceholderDirect JSON API call with stealth headers
📡 monitor_websocketBinance BTC/USDT3 WebSocket connections, 23 live messages — BTC price live
🧠 ai_extractAmazon productAI extracted name, price, rating, review count
🎯 snapshotGitHub, WikipediaARIA tree with 107/113 refs, annotated screenshots
🖱️ interactLogin flowClick → type → submit — ref targeting, session encryption, 18 action types
📦 batch_scrape10 news sitesParallel, concurrency 3, soft failure, 9/10 succeeded
📋 list_jobsBatch jobs with status and progress
📊 job_statusBatch jobFull per-URL results with timing
🗑️ delete_jobCompleted jobCleaned up job data from disk
🧠 knowledgeLocal knowledge filePer-domain stats: stealth levels, success rates, anti-bot systems detected
🎬 youtube"web scraping tutorial"Search results, video details, comments, transcripts — no API key
💬 redditr/webscrapingSubreddit posts, comments, search — public JSON API
📸 instagram@nike profileProfile details, engagement rate, recent posts — internal API
📥 downloadYouTube video, web page imagesAuto-detect URL type, download media files — images, video, og:image
📡 rssHacker News RSSParsed feed items with title, link, date, author, categories

41 tools. 34 hidden APIs on Airbnb. Live BTC feed. Reusable browser flows. Zero API keys for scraping.

Environment Variables

VariableRequiredDescription
BRAVE_API_KEYNoBrave Search API key (enables 4 search tools)
TWOCAPTCHA_API_KEYNo2Captcha API key (enables auto CAPTCHA solving)
LLM_API_KEYNoAnthropic, OpenAI, or MiniMax API key (enables ai_extract)
LLM_PROVIDERNoanthropic, openai, or minimax (default: anthropic). Recommended: minimax with MiniMax-M1 — best price/performance for extraction
LLM_MODELNoOverride default LLM model
OPENAI_API_KEYNoOpenAI API key for Whisper transcription (transcribe any YouTube video without captions)
SESSION_ENCRYPTION_KEYNo32-byte hex key for encrypting session files at rest
PROXY_URLNoSingle proxy URL (http/https/socks4/socks5)
PROXY_URLSNoComma-separated proxy URLs for rotation
BROWSER_POOL_SIZENoMax pooled browser instances (default: 3)
RESPECT_ROBOTSNoRespect robots.txt (default: true)
CHROME_PROFILE_PATHNoChrome user data dir for authenticated sessions
NO_COLORNoDisable colored output
CINoAuto-detected; disables TTY features

Development

git clone https://github.com/ceoimperiumprojects/imperium-crawl
cd imperium-crawl
npm install
npm run build
npm run dev         # Watch mode (rebuild on changes)
npm test            # 546 tests
npm start           # Start CLI (shows help or TUI)

Contributing

Contributions welcome! Whether it's a bug fix, new tool, or documentation improvement — open an issue or PR.

# Fork the repo, then:
git clone https://github.com/YOUR_USERNAME/imperium-crawl
cd imperium-crawl
npm install
git checkout -b my-feature
# Make changes...
npm test
git push origin my-feature
# Open a PR

License

MIT — use it however you want. Free forever.

Keywords

scraping

FAQs

Package last updated on 03 May 2026

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts