Map Endpoint Guide

Overview

map is the lightweight discovery tool in the stack. Use it before crawl when you need to answer:

• what URLs are reachable from this starting point,
• which subsection of the site is actually worth scraping,
• and whether a full crawl is justified at all.

curl -X POST https://fastcrw.com/api/v1/map \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"url":"https://example.com"}'

Why map Comes First

A lot of scraping waste comes from starting too broad. map helps you inspect the site structure first, which is especially useful when:

• the site has multiple product areas,
• navigation is noisy,
• or an agent needs to choose its next step instead of crawling everything.

When To Use It

Choose map when you want to answer:

• what URLs are reachable from this starting point,
• what section of the site should we target next,
• and whether a full crawl is worth the cost.

A Typical Workflow

One common pattern looks like this:

1. map a docs home page or product section.
2. Filter the returned URLs in your application.
3. Launch scrape for a handful of known-important pages.
4. Launch crawl only for the subset that deserves broader recursion.

That works well for AI agents, indexing systems, and human operators doing a first evaluation.

Output Expectations

map is for discovery, not deep extraction. Treat it as a planning primitive:

• it helps you decide what to fetch,
• it reduces unnecessary crawl scope,
• and it makes later scrape or crawl requests more intentional.

If your end goal is page content, map should usually be the first step, not the last one.