Firecrawl Integration

Overview

Firecrawl is a robust tool designed to transform websites into LLM-ready data by leveraging its Crawler and Scraper functionalities.

• Crawler: Automatically extracts data from websites by crawling web pages and following links. It recursively traverses sites (starting from a URL, using sitemaps when available), handles dynamic content rendered with JavaScript, and supports sync or async modes with webhook notifications.
• Scraper: Extracts targeted content from specific web pages using customizable rules. It supports single-URL or batch scraping, main-content-only extraction, tag inclusion/exclusion, TLS verification options, device emulation, and adjustable page-load timing.

Whether you need to map full site structures or extract specific data from pages, Firecrawl provides a seamless and customizable solution.

Features

Key Functionalities (Crawler + Scraper)

Crawler

• Comprehensive Crawling: Recursively traverses websites, identifies and accesses subpages, uses sitemaps when available, and follows links for thorough data collection.
• Dynamic Content Handling: Manages JavaScript-rendered content for full extraction from accessible subpages.
• Sync & Async Modes: Run crawls synchronously or asynchronously with webhook callbacks for completion, page, started, and failed events.

Scraper

• Customizable URL Scraping: Specify exact URLs to scrape for targeted data extraction.
• Selective Content: Scrape only main content or include/exclude specific HTML tags.
• TLS Verification: Option to skip TLS verification for broader website compatibility.
• Device Emulation: Emulate mobile devices for mobile-optimized pages.
• Adjustable Load Timing: Set wait time (ms) for dynamic content to load.

Shared

• Agent Workflows: Async Agent, Sync Agent, and Check Agent for intelligent web data extraction.
• Webhooks: Real-time updates for crawl/scrape completion and events.

Benefits

• Reliability: Handles proxies, rate limits, and anti-scraping measures for consistent extraction (Crawler).
• Efficiency: Manages requests to minimize bandwidth and avoid detection (Crawler).
• Precision: Targeted scraping with tag and scope control (Scraper).
• Flexibility: Mobile emulation and TLS options for diverse sites (Scraper).
• Generate structured, LLM-compatible data; customize inclusion/exclusion; handle static and dynamic content.

Prerequisites

Before using Firecrawl, ensure the following:

• A valid Firecrawl API Key (opens in a new tab).
• Access to the Firecrawl service host URL.
• Properly configured credentials for Firecrawl.
• A webhook endpoint for receiving notifications (required for the crawler).

Setup

Step 1: Obtain API Credentials

1. Register on Firecrawl (opens in a new tab).
2. Generate an API key from your account dashboard.
3. Note the Host URL and Webhook Endpoint.

Step 2: Configure Firecrawl Credentials

Use the following format to set up your credentials:

Configuration Reference

Sync Mode Output Format

Batched Mode

{
  "success": true,
  "status": "completed",
  "completed": 48,
  "total": 50,
  "creditsUsed": 13,
  "expiresAt": "2025-08-01T12:30:00.000Z",
  "data": [
    {
      "url": "https://example.com/page-1",
      "content": "Lorem ipsum dolor sit amet...",
      "metadata": {
        "title": "Page 1 Title",
        "description": "This is a sample description.",
        "language": "en"
      }
    },
    {
      "url": "https://example.com/page-2",
      "content": "Second page scraped content...",
      "metadata": {
        "title": "Page 2 Title",
        "description": "Another sample description.",
        "language": "en"
      }
    }
    // ... more pages
  ]
}

Single Mode

{
  "success": true,
  "status": "completed",
  "completed": 1,
  "total": 2,
  "creditsUsed": 1,
  "expiresAt": "2025-08-02T12:30:00.000Z",
  "data": [
    {
      "url": "https://example.com/page-1",
      "content": "Lorem ipsum dolor sit amet...",
      "metadata": {
        "title": "Page 1 Title",
        "description": "This is a sample description.",
        "language": "en"
      }
    }
  ]
}

Async Mode Output Format

{
  "success": true,
  "id": "8***************************7",
  "url": "https://api.firecrawl.dev/v1/crawl/8***************************7"
}

Step 3: Set Up the Mode (Crawler)

For the Crawler, choose sync or async mode. In async mode, configure a webhook endpoint to receive crawl results (create a Webhook flow/URL in Lamatic to receive crawl updates and results). Async options include:

Crawler Configuration (Single)

Batch Crawler Configuration (Async / Sync)

Scraper Configuration (Single)

Batch Scraper Configuration (Async)

Low-Code Examples

Crawler Node

nodes:
  - nodeId: crawlerNode_880
    nodeType: crawlerNode
    nodeName: Crawler
    values:
      credentials: my-firecrawl-creds
      url: https://example.com
      crawlMode: async
      webhook: https://example.com/webhook
      webhookEvents: [completed, failed, page, started]
      crawlSubPages: true
      crawlLimit: 10
      crawlDepth: 1
      excludePath: []
      includePath: []
      maxDiscoveryDepth: 1
      ignoreSitemap: false
      allowBackwardLinks: false
      allowExternalLinks: false
      delay: 0
    needs:
      - triggerNode_1

Scraper Node

nodes:
  - nodeId: scraperNode_680
    nodeType: scraperNode
    nodeName: Scraper
    values:
      credentials: my-firecrawl-creds
      url: https://example.com/page
      onlyMainContent: false
      skipTLsVerification: false
      mobile: false
      waitFor: 123
      includeTags: []
      excludeTags: []
    needs:
      - triggerNode_1

Crawler Event Output (Async Mode)

When using async crawler mode, the node can emit these events to your webhook:

1. Started: When the crawl begins (job ID, URL).
2. Page: For each crawled page (URL, extracted data).
3. Completed: When the crawl finishes (summary, page count, errors).
4. Failed: When the crawl fails (error message, job ID).

Example – Started:

{
  "success": true,
  "type": "crawl.started",
  "id": "82***********************4",
  "data": [],
  "metadata": {}
}

Scraper Output Schema

• markdown: Scraped content as Markdown.
• language: Detected language of the content.
• referrer: Referrer URL, if applicable.
• title: Title of the scraped page.
• scrapeId: Unique identifier for the scrape.
• sourceURL: URL from which content was scraped.
• url: Resolved URL of the resource.
• statusCode: HTTP status code of the scrape.

Example:

{
  "markdown": "# Page content...",
  "language": "en",
  "referrer": "",
  "title": "Example Page",
  "scrapeId": "uuid-here",
  "sourceURL": "https://example.com/page",
  "url": "https://example.com/page",
  "statusCode": 200
}

Map URL Configuration

Map URL Output Example

{
  "success": true,
  "links": [
    "https://lamatic.ai/docs",
    "https://lamatic.ai/docs/architecture",
    "https://lamatic.ai/docs/career",
    "https://lamatic.ai/docs/context",
    "https://lamatic.ai/docs/context/vectordb",
    "https://lamatic.ai/docs/context/vectordb/adding-data",
    "https://lamatic.ai/docs/contributing",
    "https://lamatic.ai/docs/demo",
    "https://lamatic.ai/docs/deployments",
    "https://lamatic.ai/docs/deployments/cache"
  ]
}

Agent Workflow Configuration

Firecrawl now supports agent workflows with three execution modes for submitting and monitoring agent tasks.

Agent Modes

1. Async Agent: Submit agent tasks asynchronously and receive a task ID for later status checking.
2. Sync Agent: Execute agent tasks synchronously and receive results immediately upon completion.
3. Check Agent: Check the status of a previously submitted async agent task.

Agent Configuration (Async / Sync)

Check Agent Configuration

Troubleshooting

Common Issues

Debugging

• Check Firecrawl logs for detailed error information.
• Test the webhook endpoint to confirm it is receiving updates.
• If the connection fails, whitelist the following IPs: https://www.cloudflare.com/ips/ (opens in a new tab)