Web Page to Clean Markdown for RAG & LLM — 77% Fewer Tokens

Turn any web page into clean Markdown for your LLM or RAG pipeline — using 77% fewer tokens than a raw crawl, while keeping every content section intact.

Works in any language. No browser overhead.

Why this matters

Most crawlers hand your LLM the whole page: navigation, cookie banners, sidebars, footers, "Jump to content" links, image markup. You pay for every one of those tokens, on every page, on every run.

This Actor strips the page down to what your model actually needs.

Measured, not claimed

Wikipedia article Retrieval-augmented generation — same URL, same day, links enabled on both sides. Named baselines, so you can reproduce this:

	This Actor	RAG Web Browser	Website Content Crawler
Characters returned	16 760	73 477	29 533
Approx. input tokens	~4 190	~18 369	~7 383
Headings preserved	14	14	3

Two honest conclusions:

• Against a browser-based crawl that keeps page furniture (RAG Web Browser): 77% fewer tokens for the same content sections.
• Against a well-tuned HTML crawler (Website Content Crawler): token counts are roughly equal — but that crawler flattens the document to 3 headings, while this Actor keeps all 14.

Why headings matter more than they look

RAG pipelines chunk documents. Headings are the natural chunk boundaries — they carry the section topic into the embedding. A document flattened to 3 headings gives your splitter almost nothing to work with, so chunks end up cut mid-argument and retrieval quality drops.

Structure is not cosmetic here. It is what makes the extracted text usable.

What token savings mean on a real workload

Processing 100 000 pages per month, against a browser-based crawl:

	This Actor	Browser-based crawl
Tokens sent to your LLM	0.42 B	1.84 B
Cost at $1 / M input tokens	~$420	~$1 840

~$1 400/month saved on token spend alone — before counting compute.

What you get

Input:

{
  "urls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }],
  "outputFormats": ["markdown"]
}

Output (one record per URL):

{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "finalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "status": "ok",
  "httpStatus": 200,
  "title": "Retrieval-augmented generation - Wikipedia",
  "description": "",
  "language": "en",
  "author": "",
  "published": "",
  "siteName": "Wikipedia",
  "wordCount": 2383,
  "reductionRatio": 0.12,
  "markdown": "# Retrieval-augmented generation - Wikipedia\n\nFrom Wikipedia, the free encyclopedia\n\nType of information retrieval using LLMs\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources...\n\n## RAG and LLM limitations\n\n...",
  "links": []
}

Note how the Markdown opens directly with the title and first sentence — no navigation preamble to pay for.

Any language

Structural extraction only — no keyword or language assumptions. Verified on English, Polish, German, Japanese and French Wikipedia in a single run:

Language	Words extracted	Size vs original
English	3 176	12.0%
Polish	2 831	17.3%
German	7 854	26.3%
Japanese	5 578	16.6%
French	18 124	26.6%

The language field is returned per page, so you can route or filter downstream.

How to use

1. Paste one or more URLs into URLs.
2. Leave Output formats as markdown (most token-efficient) — or add text / html.
3. Run. Each page becomes one dataset record.

Set Max characters per page to hard-cap your token budget, or turn off Keep links to strip link markup and save even more.

From code

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("craftbox/fast-markdown-for-rag").call(run_input={
    "urls": [{"url": "https://example.com/article"}],
    "outputFormats": ["markdown"],
    "keepLinks": False,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["markdown"])

Pricing

$1.50 per 1 000 pages processed. You are charged only for pages returned with status: ok — failed and skipped URLs are free.

Compute on top of that is roughly $0.21 per 1 000 pages, because this Actor uses plain HTTP requests instead of a headless browser.

Total: about $1.71 per 1 000 pages — and that is before the token savings above.

Limitations — read this before you run

Stated plainly, so there are no surprises:

1. JavaScript is not executed. Pages rendered entirely client-side (many SPAs) will return little content. When that happens, the record is flagged with requiresJavascript: true and a warning — rather than quietly returning a thin result as a success. Check that field.
2. No search. You provide URLs; this Actor does not query search engines.
3. HTTP errors are failures, not empty content. A 404 or 500 returns status: "failed" with the status code, never a blank page marked as OK.
4. Rate limiting is on by default (1 second per host). Raise concurrency or lower the delay only where the source permits it.

Politeness and compliance

This Actor fetches only publicly accessible pages. It does not log in, does not bypass any protection mechanism, and enforces a per-host delay by default. You are responsible for ensuring your use of the extracted content complies with the source's terms and with applicable law.

FAQ

How does it compare to a headless-browser crawler? For static and server-rendered pages, this returns the same content sections with far fewer tokens and roughly 10× lower compute cost. For client-rendered pages, a browser-based crawler will extract more — use one of those instead, and see limitation 1.

How does it compare to Website Content Crawler? Straight answer: on token count they are about equal, and that Actor is cheaper per page, crawls whole sites and handles JavaScript. Pick this one when you fetch known URLs and care about document structure — it keeps 14 headings where that crawler keeps 3, which matters if you chunk by heading. Pick that one for whole-site crawls or JS-heavy pages.

Will it break my Markdown structure? Headings, lists, tables, emphasis and links are preserved. Images are dropped by design, as image markup is pure token cost in a RAG context.

Can I get plain text instead? Yes — add text to outputFormats. You can request several formats in one run.

How do I keep output small for a tight context window? Set maxChars to your budget and keepLinks to false. Records that get cut are marked with truncated: true.

Does it handle redirects? Yes. The resolved address is returned in finalUrl.

What if a source rate-limits me? Enable a proxy in the input, raise perHostDelay, or lower maxConcurrency. Temporary failures (429, 503) are retried with exponential backoff.