How to Scrape Websites into LLM-Ready Markdown (RAG Guide)

If you're building anything with LLMs — a RAG pipeline, a research agent, a competitive-intelligence dashboard — you eventually hit the same wall: the content you need lives on web pages, and web pages are full of garbage. Navigation bars, cookie banners, ads, footers, and sidebars all end up in your context window, burning tokens and confusing retrieval. The fix is converting pages to clean Markdown before they touch your model. Here's how, from DIY to one API call.

Why Markdown is the right format for LLMs

LLMs read Markdown natively: headings preserve document structure, lists stay lists, links keep their targets, and tables survive. Compared to raw HTML, Markdown of the same page is typically 80–95% smaller — which means cheaper embeddings, more room in the context window, and better retrieval, because your chunks contain content instead of <div class="nav-wrapper">.

The hard part isn't the format conversion. It's boilerplate removal: deciding which parts of the page are the article and which are the chrome around it.

Option 1: DIY with Python

The open-source stack is httpx + trafilatura (the same extraction engine many commercial tools build on):

import httpx
import trafilatura

html = httpx.get("https://example.com/article").text
markdown = trafilatura.extract(html, output_format="markdown", include_links=True)
print(markdown)

This gets you surprisingly far for well-behaved sites. Where it breaks down in production:

• Blocking. Many sites 403 datacenter IPs; you'll need proxy rotation.
• Coverage. JS-heavy pages and unusual layouts defeat pure-HTML extraction.
• Crawling. Turning "this page" into "this whole docs site" means writing a link-discovery + queue + dedupe layer yourself.

Option 2: one API call per page

A managed website-to-Markdown API handles fetching (through residential proxies), extraction, and metadata in one request:

curl -X POST https://transcriptfetch.com/api/v1/web \
  -H "Authorization: Bearer tf_live_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article"}'

{
  "ok": true,
  "data": {
    "kind": "web",
    "title": "Example article",
    "markdown": "# Example article\n\nClean, boilerplate-free content ...",
    "metadata": { "author": "Jane Doe", "date": "2026-01-15" }
  },
  "usage": { "credits_spent": 1 }
}

The metadata matters more than it looks: author and date become filterable attributes in your vector store, which is how you answer "what did they publish this year?" without re-reading everything.

Scaling up: map, then crawl

For whole sites — a docs site into your RAG index, a competitor's blog into your research corpus — two more endpoints complete the workflow:

# 1. Discover every URL on the site (page links + sitemap.xml)
curl -X POST https://transcriptfetch.com/api/v1/web/map \
  -d '{"url": "https://example.com"}' ...

# 2. Crawl a section: same-site BFS, clean Markdown per page
curl -X POST https://transcriptfetch.com/api/v1/web/crawl \
  -d '{"url": "https://example.com/docs", "limit": 10}' ...

Crawls bill one credit per page actually returned — a page that blocks or has no readable content costs nothing. See the web scraping docs for the full reference.

Feeding it into a RAG pipeline

The end-to-end shape most teams land on:

1. Map the site → list of URLs.
2. Crawl/scrape the URLs you care about → Markdown + metadata.
3. Chunk by Markdown headings (structure-aware splits beat fixed-size ones).
4. Embed + store with url, title, author, date as metadata.
5. Re-scrape on a schedule; the date field tells you what actually changed.

And if your consumer is an agent rather than a pipeline: the same capability is exposed over MCP as scrape_url and map_url, so Claude or Cursor can read pages mid-conversation — same key, same credits.

One integration also covers the video side — YouTube, TikTok, Instagram, X, and Facebook transcripts come from the same API — so "ingest everything this company publishes" is one key instead of five tools. Start with 100 free credits, no card required.