n8n-nodes-firecrawl-scraper

n8n-nodes-firecrawl-scraper

This package contains n8n nodes for working with the Firecrawl API, allowing you to scrape, crawl, and map websites directly within your n8n workflows. It serves as a wrapper around the Firecrawl npm SDK, primarily focusing on the extract feature.

Note: This project is still in development. Features and functionality may change.

Support and Feedback

For support, feedback, or to contribute to this project:

• GitHub Repository: n8n-nodes-firecrawl-scraper
• Tutorial Video: Watch on YouTube

Prerequisites

• A Firecrawl API key (get one from firecrawl.dev)
• n8n instance (v1.0.0 or later)

Installation

Follow these steps to install this custom node package in your n8n instance:

In an existing n8n instance

• Go to Settings > Community Nodes
• Select Install
• Enter n8n-nodes-firecrawl-scraper in the Name field
• Agree to the risks of using community nodes
• Click Install

Manually (Development)

• Clone this repository
• Navigate to the directory: cd n8n-nodes-firecrawl-scraper
• Install dependencies: npm install
• Build the code: npm run build
• Link to your n8n installation: npm link
• In your n8n installation directory, run: npm link n8n-nodes-firecrawl-scraper

Nodes

This package includes a single Firecrawl node with multiple operations:

Firecrawl

A unified node that provides various operations to interact with the Firecrawl API:

Operations:

• Scrape: Scrape content from a single URL in Markdown format
• Crawl: Crawl multiple pages of a website and retrieve content in Markdown format
• Map: Generate a list of URLs from a website
• Extract: Extract structured data from URLs using simple extraction or schema-based extraction

Common Parameters:

• URL: Required for all operations - specifies the target URL(s)
• Enable Debug Logs: Available for Scrape and Extract operations - enables detailed logging

Operation-Specific Parameters:

• Crawl:

  • Limit: Maximum number of pages to crawl (default: 50)

• Extract:

  • Extraction Method: Simple extraction or Schema-based extraction
  • Extraction Prompt: Guide the extraction process
  • Schema Definition Type (for schema-based extraction): Generate from JSON example or define manually
  • JSON Example or Schema Definition: Define the data structure

Credentials

You'll need to set up your Firecrawl API key in the Firecrawl API credential type:

• In your n8n workflow, add any Firecrawl node
• Click the Create new credential button
• Enter your Firecrawl API key
• Save the credential

Examples

Basic Web Scraping

• Add a Firecrawl node
• Select the Scrape operation
• Set the URL to the webpage you want to scrape
• Connect to nodes like ChatGPT or Text processors to analyze the scraped content

Website Crawling for Data Extraction

• Add a Firecrawl node
• Select the Crawl operation
• Set the URL to the website you want to crawl
• Set a limit (e.g., 10 pages)
• Connect to database nodes to store the extracted data

Website Mapping for SEO Analysis

• Add a Firecrawl node
• Select the Map operation
• Set the URL to the website you want to map
• Connect to spreadsheet or visualization nodes to analyze the site structure

Structured Data Extraction

• Add a Firecrawl node
• Select the Extract operation
• Set the URL(s) to extract data from
• Choose the extraction method (Simple or Schema-based)
• If using Schema-based extraction, define your schema or provide a JSON example
• Use the extracted structured data in your workflow

License

MIT