CrewAI Web Scraping Integration — fastCRW [Firecrawl-Compatible]

Why CrewAI + fastCRW

CrewAI organizes multi-agent workflows around explicit roles, tasks, and tools. Most CrewAI crews need at least one agent that pulls live web context, and that scrape tool ends up on the critical path of the entire crew. fastCRW is the lightest viable scraping primitive for CrewAI — a 6.6 MB RAM runtime that returns clean Markdown so the agent's context window stays focused on reasoning, not HTML cleanup. Because fastCRW is Firecrawl-compatible, CrewAI tools written for Firecrawl port directly to fastCRW with only a base URL change.

Setup

1. Install CrewAI and the standard tools package.
2. Provision a fastCRW API key from the dashboard.
3. Export FASTCRW_API_KEY in the environment your CrewAI process reads.
4. Define a BaseTool subclass that posts to the fastCRW scrape and search endpoints.

pip install -U crewai crewai-tools requests
export FASTCRW_API_KEY="fcrw_..."

You do not need a CrewAI-specific fastCRW package. The standard BaseTool interface is enough.

Code Example

import os
import requests
from typing import Type
from pydantic import BaseModel, Field
from crewai import Agent, Task, Crew
from crewai.tools import BaseTool

FASTCRW_BASE = "https://api.fastcrw.com"

class FastCRWScrapeInput(BaseModel):
    url: str = Field(..., description="The URL to scrape.")

class FastCRWScrapeTool(BaseTool):
    name: str = "fastcrw_scrape"
    description: str = "Scrape a URL via fastCRW and return clean Markdown."
    args_schema: Type[BaseModel] = FastCRWScrapeInput

    def _run(self, url: str) -> str:
        r = requests.post(
            f"{FASTCRW_BASE}/v1/scrape",
            headers={
                "Authorization": f"Bearer {os.environ['FASTCRW_API_KEY']}",
            },
            json={"url": url, "formats": ["markdown"]},
            timeout=60,
        )
        r.raise_for_status()
        return r.json()["data"]["markdown"]

class FastCRWSearchInput(BaseModel):
    query: str = Field(..., description="Web search query.")

class FastCRWSearchTool(BaseTool):
    name: str = "fastcrw_search"
    description: str = "Search the live web via fastCRW. Returns ranked results."
    args_schema: Type[BaseModel] = FastCRWSearchInput

    def _run(self, query: str) -> list[dict]:
        r = requests.post(
            f"{FASTCRW_BASE}/v1/search",
            headers={
                "Authorization": f"Bearer {os.environ['FASTCRW_API_KEY']}",
            },
            json={"query": query, "limit": 5},
            timeout=60,
        )
        r.raise_for_status()
        return r.json()["data"]

scrape = FastCRWScrapeTool()
search = FastCRWSearchTool()

researcher = Agent(
    role="Web research analyst",
    goal="Pull primary sources via fastCRW",
    backstory="Skilled at distilling raw web pages into structured findings.",
    tools=[search, scrape],
)

task = Task(
    description="Research the latest open-source web scraping APIs.",
    expected_output="A bulleted summary with citations.",
    agent=researcher,
)

crew = Crew(agents=[researcher], tasks=[task])
crew.kickoff()

When to Use This

• Research crews — one CrewAI agent searches, another scrapes top results, a third writes the report.
• Lead enrichment — a CrewAI sales agent calls fastCRW to scrape company sites before drafting outreach.
• Competitive monitoring — schedule a CrewAI crew to scrape competitor pages on a cron via fastCRW.
• Migrating Firecrawl-based crews — keep the crew topology and only swap the tool implementation to fastCRW.

Limits + Gotchas

• CrewAI tools run synchronously by default. Long fastCRW crawls block the agent — prefer scrape and search for in-loop work.
• Tool errors raised inside _run propagate up. Catch HTTP errors and return an explanatory string so the agent can recover.
• The description field on each tool is critical — CrewAI agents pick tools based on the description text. Be precise about when to use fastCRW.
• CrewAI does not deduplicate tool calls. Cache repeated fastCRW URLs at your application layer to avoid wasted scrapes.

Related

• LangChain integration
• LangGraph integration
• fastCRW search docs
• Firecrawl alternative