Guides

Web scraping

Three endpoints: scrape one page to Markdown, map a site's links, or crawl whole sections page by page.

POST /api/v1/web — one URL in, clean Markdown out. Navigation, ads, and boilerplate removed; title/author/date metadata included. 1 credit per page.

POST /api/v1/web/map — every same-site link on a page plus its sitemap.xml. Use it to discover what’s on a site before scraping. 1 credit per call.

POST /api/v1/web/crawl — breadth-first crawl across same-site links, Markdown per page, up to 25 pages per call. 1 credit per page returned.

Scrape a page

curl -X POST https://transcriptfetch.com/api/v1/web \
  -H "Authorization: Bearer tf_live_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article"}'

{
  "ok": true,
  "data": {
    "kind": "web",
    "url": "https://example.com/article",
    "title": "Example article",
    "markdown": "# Example article\n\nClean content ...",
    "metadata": { "author": "Jane Doe", "date": "2026-01-15", "sitename": "Example" }
  },
  "usage": { "credits_spent": 1, "bytes": 48211 }
}

Map, then crawl

# 1. Discover the site's URLs (page links + sitemap)
curl -X POST https://transcriptfetch.com/api/v1/web/map \
  -H "Authorization: Bearer tf_live_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "limit": 200}'

# 2. Crawl a section to Markdown (max 25 pages/call; 1 credit per page)
curl -X POST https://transcriptfetch.com/api/v1/web/crawl \
  -H "Authorization: Bearer tf_live_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/docs", "limit": 10}'

Failures aren’t charged. A page that blocks the request, returns no readable content, or errors costs 0 credits — usage.credits_spent always reports what was actually billed. Crawls bill only the pages returned in data.pages.

Agents: the MCP server exposes scrape_url and map_url, so assistants in Claude, Cursor, or ChatGPT can read web pages mid-conversation. For whole-site work, agents map first and scrape the pages they need.

Next →Rate limits