Flowise Web Scraping Integration — fastCRW [Firecrawl-Compatible]

Why Flowise + fastCRW

Flowise is the drag-and-drop interface for building LangChain applications without coding. It turns Claude, GPT, and other LLMs into production AI agents by orchestrating prompts, retrievers, tools, and memory. The missing link is web scraping — Flowise can ingest files and knowledge bases, but fetching live web data requires switching to code or calling external services. fastCRW fits inside Flowise through the HTTP Request node or a custom tool definition, enabling you to scrape, search, and extract web data in a visual workflow. The 6.6 MB RAM fastCRW runtime is lightweight and self-hostable, pairing naturally with self-hosted Flowise deployments.

Setup: HTTP Request Node

Flowise ships with a built-in HTTP Request node. Use it to call fastCRW:

1. Open Flowise and create or open a flow.
2. Add an HTTP Request node from the toolbox (search for "http").
3. Set Method to POST.
4. Set URL to https://fastcrw.com/api/v1/scrape (or /search, /crawl, /map depending on your use case).
5. Add a Header:
   • Key: Authorization
   • Value: Bearer fcrw_YOUR_API_KEY (store the key as a Flowise credential or environment variable)
6. Set Content-Type to application/json.
7. In the Body field, add JSON:

{
  "url": "{{ url }}",
  "formats": ["markdown"]
}

Connect an upstream node that provides the url variable (e.g., a Trigger, Text Input, or Code node).

Setup: Custom Tool (Advanced)

For more control, define a Custom Tool in your Flowise flow:

1. Add a Custom Tool node from the toolbox.
2. Set Tool Name to fastcrawl_scrape.
3. Set Tool Description to "Scrape a URL using fastCRW and return markdown content."
4. Add Tool Input Parameters (JSON):

{
  "url": {
    "type": "string",
    "description": "The URL to scrape"
  },
  "format": {
    "type": "string",
    "description": "Output format: markdown, html, or raw",
    "default": "markdown"
  }
}

5. Set Tool Function to JavaScript that calls fastCRW:

async function fastcrawl_scrape(url, format = "markdown") {
  const response = await fetch("https://fastcrw.com/api/v1/scrape", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.FASTCRW_API_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      url: url,
      formats: [format]
    })
  });
  const data = await response.json();
  return data.data[format] || data.data.markdown;
}

6. Connect the Custom Tool to your LLM or downstream nodes.

Code Example: RAG with Flowise

A complete Flowise flow for RAG (Retrieval-Augmented Generation) with fastCRW:

Flow structure:

1. Trigger (accepts user input: URL to ingest)
2. HTTP Request (calls fastCRW scrape, fetches markdown)
3. Text Splitter (chunks the markdown into 512-token pieces)
4. Embeddings (generates vectors — use OpenAI, Hugging Face, or local)
5. Vector Store (Pinecone, Weaviate, Supabase, or in-memory)
6. Output (confirmation message)

In the HTTP Request node body:

{
  "url": "{{ trigger.url }}",
  "formats": ["markdown"],
  "timeout": 30
}

The HTTP node outputs an object with a data property containing markdown. Connect the output to the Text Splitter and set the input field to {{ http_request.data.markdown }}.

For inference (querying the RAG):

1. User Input (query text)
2. Retriever (searches the vector store with the query)
3. Prompt Template (composes: system prompt + retrieved docs + user query)
4. LLM (Claude, GPT, etc.)
5. Output (AI response)

This Flowise flow now retrieves grounded answers from scraped web pages.

Code Example: AI Agent with fastCRW

A Flowise agent that can scrape web pages autonomously:

Flow structure:

1. Trigger (user message)
2. Memory (conversation history)
3. Chat Prompt Template (system prompt for the agent)
4. LLM (Claude, GPT, etc.)
5. Agent Tools (define fastCRW scrape as a tool)
6. Agent Loop (LLM decides when to call tools)
7. Output (agent's final response)

Define fastCRW as a Custom Tool with:

async function fastcrawl_scrape(url) {
  const response = await fetch("https://fastcrw.com/api/v1/scrape", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.FASTCRW_API_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({ url, formats: ["markdown"] })
  });
  const result = await response.json();
  return result.data.markdown;
}

Add this tool to the Agent Tools node. Now when the user asks "What is the pricing on example.com?", the LLM can invoke fastCRW autonomously, scrape the page, and summarize the pricing.

Advanced: Async Crawl with Polling

For large crawls that may timeout:

1. POST to https://fastcrw.com/api/v1/crawl with "async": true:

{
  "url": "https://example.com",
  "async": true,
  "max_depth": 3,
  "max_pages": 50
}

2. Capture the response job_id.
3. Use a Loop node to poll https://fastcrw.com/api/v1/crawl/{job_id} until status is completed.
4. Fetch the final results and continue your flow.

Flowise doesn't have a native polling loop, so implement this with a Code node and setInterval.

When to Use This

• No-code RAG pipelines — scrape web pages, index them, and query them in a visual flow.
• AI agent data fetching — give Claude or GPT a fastCRW tool so it can browse the web during conversations.
• Dynamic knowledge base — schedule a Flowise flow to periodically scrape and re-index a knowledge base.
• Lead enrichment — scrape company websites and feed the data into an LLM for enrichment and scoring.
• Competitive intelligence — build an agent that scrapes competitor sites and generates reports.

Troubleshooting

"Bearer token missing" Ensure the HTTP Request node has an Authorization header with value Bearer fcrw_YOUR_API_KEY. If using an environment variable, set FASTCRW_API_KEY in your Flowise .env file or hosting platform's environment.

"Request timeout" The default HTTP timeout in Flowise is 30 seconds. For slower pages, increase the timeout in the HTTP node configuration. Alternatively, use async crawl mode with polling.

"404 on Custom Tool" Make sure the Custom Tool function is syntactically correct and process.env.FASTCRW_API_KEY is available in your Flowise runtime. Self-hosted Flowise may need environment variables explicitly set in the startup script.

"Vector store not indexing results" Verify that the Text Splitter node is receiving markdown from the HTTP Request. Check the flow's Variables inspector to see what data is being passed between nodes. The Vector Store node expects chunked text with metadata.

"Agent not calling the scrape tool" The agent calls tools only if its system prompt mentions them and the LLM decides they're relevant. Make sure the Custom Tool is added to the Agent Tools list and the system prompt says the agent has access to a scraping tool.

When to Choose fastCRW

• Speed and lightweight runtime: fastCRW is 5.5x faster and 75x lighter on memory than Firecrawl, making it ideal for self-hosted Flowise.
• Firecrawl compatibility: Flowise flows built on Firecrawl's HTTP API port to fastCRW by changing one URL.
• Self-hosting: fastCRW's single binary runs on a $5 VPS or Docker container alongside self-hosted Flowise.
• Cost: Consumption-based pricing means you pay only for what you scrape, not for idle infrastructure.
• MCP support: If you also use Claude Code or other MCP clients, fastCRW provides a unified scraping API across tools.

Related

• n8n integration
• Dify integration
• Make integration
• Claude Code integration
• fastCRW scrape docs
• fastCRW search docs
• Flowise documentation
• LangChain tools guide