Web Scraping for LLM Agents

Who this is for

Teams building LLM agents — research copilots, customer-support bots, internal "ask my data" assistants — that need a reliable way to put live web content in the model's context window. An agent without a browse tool is a chatbot with a 2024 cut-off; one with a flaky browse tool is worse, because it confidently cites pages that timed out.

fastCRW gives the agent two clean tool surfaces — search and scrape — that return predictable markdown the model can reason over.

Why fastCRW for agents

Agents care about three things in a browse tool: the call returns, it returns fast enough that the planner does not give up, and the payload is small enough that the next turn's tokens stay cheap. fastCRW is built around those three constraints.

POST /v1/search (docs.fastcrw.com/api-reference/search/) returns web results with optional inline content scraping, so the agent can go from query to grounded text in one tool call. POST /v1/scrape (docs.fastcrw.com/api-reference/scrape/) takes a single URL and hands back clean markdown. Both endpoints share the Firecrawl response envelope, so an agent already wired to Firecrawl's /v1/scrape only needs a base-URL change.

For MCP-aware hosts (Claude Desktop, Cursor, Cline, Windsurf), the crw-mcp@0.6.0 package (dist-tag latest on npm, per marketing/CANONICAL-FACTS.md) exposes the same endpoints as MCP tools — no custom transport code on the agent side.

The 5-step recipe

1. Wire fastCRW as a tool the agent can call. Expose two tools to the agent — search(query) and scrape(url) — backed by POST /v1/search and POST /v1/scrape. If your agent host speaks MCP, point it at crw-mcp@0.6.0 and skip the REST wiring entirely.
2. Let the planner pick search vs scrape. In the system prompt, tell the model to call search when it needs to discover URLs and scrape when it already has one. Cap iterations so a confused plan cannot loop the tool indefinitely.
3. Run the tool call and return markdown. Execute the tool against fastCRW. Return only the markdown body and the source URL to the model — not raw HTML, not full response envelopes. Smaller tool outputs mean cheaper next-turn tokens.
4. Let the model reason over the retrieved content. Feed the tool result back as a tool message. The agent now has grounded text it can quote, summarise, or extract structured fields from.
5. Cite the source in the final answer. Force the agent to surface the source URL beside any claim it makes. Auditable answers beat confident hallucinations every time.

# agent_loop.py — run with: python3 agent_loop.py
import os
import json
import requests
from anthropic import Anthropic

CRW = "https://api.fastcrw.com/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['CRW_API_KEY']}"}
claude = Anthropic()

TOOLS = [
    {
        "name": "search",
        "description": "Search the web. Returns a list of {title, url, snippet}.",
        "input_schema": {
            "type": "object",
            "properties": {"query": {"type": "string"}},
            "required": ["query"],
        },
    },
    {
        "name": "scrape",
        "description": "Fetch a URL as clean markdown. Use only after search.",
        "input_schema": {
            "type": "object",
            "properties": {"url": {"type": "string"}},
            "required": ["url"],
        },
    },
]

def run_tool(name: str, args: dict) -> str:
    if name == "search":
        r = requests.post(f"{CRW}/search", json={"query": args["query"], "limit": 5},
                          headers=HEADERS, timeout=30)
        return json.dumps(r.json()["data"]["results"])
    if name == "scrape":
        r = requests.post(f"{CRW}/scrape", json={"url": args["url"], "formats": ["markdown"]},
                          headers=HEADERS, timeout=60)
        return r.json()["data"]["markdown"][:8000]
    raise ValueError(name)

def chat(user_prompt: str, max_turns: int = 4) -> str:
    messages = [{"role": "user", "content": user_prompt}]
    for _ in range(max_turns):
        resp = claude.messages.create(
            model="claude-sonnet-4-6", max_tokens=1024, tools=TOOLS, messages=messages,
        )
        if resp.stop_reason != "tool_use":
            return resp.content[0].text
        tool_use = next(b for b in resp.content if b.type == "tool_use")
        tool_result = run_tool(tool_use.name, tool_use.input)
        messages += [
            {"role": "assistant", "content": resp.content},
            {"role": "user", "content": [{
                "type": "tool_result", "tool_use_id": tool_use.id, "content": tool_result,
            }]},
        ]
    return "Tool budget exhausted."

if __name__ == "__main__":
    print(chat("What is the p50 scrape latency reported in the fastCRW 3-way benchmark?"))

Next steps

The MCP install path and tool reference live at docs.fastcrw.com; pricing for managed-API agent traffic is on fastcrw.com/pricing. Self-host the same binary for $0 per 1,000 scrapes under AGPL-3.0 when your agent runs inside a private network.