OpenAI Agents SDK Web Scraping Integration — fastCRW [Firecrawl-Compatible]

Why OpenAI Agents SDK + fastCRW

The OpenAI Agents SDK is the official path for building agentic systems against OpenAI models. Tools are the primitive that connects an Agent to the real world, and web scraping is one of the most common tools any production agent needs. fastCRW is the right scraping primitive for the OpenAI Agents SDK because the API is Firecrawl-compatible and the runtime is 6.6 MB of RAM, which keeps tool latency low and tracing readable. Wrap fastCRW once with @function_tool and every Agent in your system — including multi-agent Handoff graphs — can scrape, search, and extract live pages without bespoke glue.

Setup

1. Install the OpenAI Agents SDK and requests.
2. Provision a fastCRW API key from the dashboard.
3. Export FASTCRW_API_KEY and OPENAI_API_KEY in your environment.
4. Define @function_tool-decorated functions that wrap the fastCRW scrape and search endpoints.

pip install -U openai-agents requests
export FASTCRW_API_KEY="fcrw_..."
export OPENAI_API_KEY="sk-..."

The OpenAI Agents SDK does not require a custom integration package for fastCRW. Any function decorated with @function_tool becomes available to the Agent.

Code Example

import os
import requests
from agents import Agent, Runner, function_tool

FASTCRW_BASE = "https://api.fastcrw.com"

@function_tool
def fastcrw_scrape(url: str) -> str:
    """Scrape a URL via fastCRW and return Markdown."""
    r = requests.post(
        f"{FASTCRW_BASE}/v1/scrape",
        headers={"Authorization": f"Bearer {os.environ['FASTCRW_API_KEY']}"},
        json={"url": url, "formats": ["markdown"]},
        timeout=60,
    )
    r.raise_for_status()
    return r.json()["data"]["markdown"]

@function_tool
def fastcrw_search(query: str, limit: int = 5) -> list[dict]:
    """Web search via fastCRW. Returns ranked results."""
    r = requests.post(
        f"{FASTCRW_BASE}/v1/search",
        headers={"Authorization": f"Bearer {os.environ['FASTCRW_API_KEY']}"},
        json={"query": query, "limit": limit},
        timeout=60,
    )
    r.raise_for_status()
    return r.json()["data"]

researcher = Agent(
    name="Web researcher",
    instructions=(
        "Use fastcrw_search to find sources, then fastcrw_scrape "
        "to read the top results before answering."
    ),
    tools=[fastcrw_search, fastcrw_scrape],
)

result = Runner.run_sync(researcher, "Summarize the latest changes to the Anthropic API.")
print(result.final_output)

For a multi-agent system, define a separate writer Agent with a Handoff from researcher so the research agent collects fastCRW results and hands the synthesis to the writer. The OpenAI Agents SDK preserves the trace across the handoff so every fastCRW tool call shows up in one timeline.

When to Use This

• Research agents — search the web with fastCRW, scrape the top hits, then summarize.
• Customer support agents — pull product pages or status pages on demand to ground responses.
• Competitive intelligence — schedule an OpenAI Agents SDK runner to monitor competitor pricing pages.
• Multi-agent Handoff graphs — let one agent gather web context via fastCRW and hand off to a writer or analyst.

Limits + Gotchas

• The OpenAI Agents SDK enforces a per-run iteration limit. Long fastCRW crawls can blow that limit — prefer scrape per URL inside the loop.
• Tool errors raised inside @function_tool functions surface to the Agent. Catch HTTP errors and return descriptive strings so the Agent can recover gracefully.
• The SDK's tracing UI captures full tool inputs and outputs. Avoid scraping pages that contain secrets — they will show up in the trace.
• Function tool argument typing must be JSON-serializable. Keep fastCRW tool signatures to primitive types and basic dicts.

Related

• LangChain integration
• Google ADK integration
• fastCRW search docs
• Firecrawl alternative