fcrawl

🔥 fcrawl

Elegant Firecrawl CLI wrapper for web crawling and content extraction

A powerful command-line tool for crawling websites and extracting content using the Firecrawl API. Easily save web content in multiple formats including Markdown, JSON, and more.

✨ Features

• 🚀 Fast and efficient web crawling
• 📂 Save content in multiple formats (Markdown, JSON, etc.)
• 🔍 Extract specific content using natural language prompts
• ⚡ Resume interrupted crawls
• 📊 Progress tracking and detailed logging
• 🛠️ Configurable with sensible defaults

🚀 Installation

# Install globally
npm install -g fcrawl

# Or use with npx
npx fcrawl <url> [options]

📋 Usage

# Basic usage
fcrawl https://example.com

# With options
fcrawl https://example.com --limit 100 --output ./output --md

# Resume an interrupted crawl
fcrawl https://example.com --resume

Options

  -v, --version           Output the version number
  -l, --limit <number>    Maximum number of pages to crawl (default: 1000)
  -d, --max-depth <n>     Maximum crawl depth (default: 10)
  -o, --output <dir>      Output directory (defaults to domain name)
  --md                    Convert PDFs and documents to markdown
  -f, --formats <list>    Output formats (comma-separated) (default: "markdown")
  --include-paths <p>     URL patterns to include (comma-separated)
  --exclude-paths <p>     URL patterns to exclude (comma-separated)
  --extract <prompt>      Natural language extraction prompt
  --delay <ms>            Milliseconds to wait between requests
  --timeout <ms>          Request timeout in milliseconds (default: 30000)
  --resume                Resume interrupted crawl
  --mobile                Use mobile user agent
  --no-block-ads          Disable ad blocking
  -h, --help              Display help for command

🔧 Development

• Clone the repository:

  git clone https://github.com/chadananda/fcrawl.git
  cd fcrawl
  

• Install dependencies:

  npm install
  

• Set up your environment variables:

  cp .env.example .env
  # Edit .env with your API keys
  

• Run in development mode:

  npm run dev
  

🧪 Testing

# Run tests
npm test

# Run tests in watch mode
npm test -- --watch

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

This project is licensed under the ISC License - see the LICENSE file for details.

Made with ❤️ by Chad Jones