# Web Page to Clean Markdown for RAG & LLM — 77% Fewer Tokens (`craftbox/fast-markdown-for-rag`) Actor

Convert web pages into clean Markdown for LLM and RAG pipelines, using 77% fewer tokens than a raw crawl while keeping every content section. Works in any language.

- **URL**: https://apify.com/craftbox/fast-markdown-for-rag.md
- **Developed by:** [Craftbox](https://apify.com/craftbox) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.50 / 1,000 page processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Web Page to Clean Markdown for RAG & LLM — 77% Fewer Tokens

Turn any web page into clean Markdown for your LLM or RAG pipeline — using **77% fewer
tokens** than a raw crawl, while keeping **every content section** intact.

Works in any language. No browser overhead.

### Why this matters

Most crawlers hand your LLM the whole page: navigation, cookie banners, sidebars, footers,
"Jump to content" links, image markup. You pay for every one of those tokens, on every
page, on every run.

This Actor strips the page down to what your model actually needs.

#### Measured, not claimed

Wikipedia article *Retrieval-augmented generation* — same URL, same day, links enabled
on both sides. Named baselines, so you can reproduce this:

| | This Actor | RAG Web Browser | Website Content Crawler |
|---|---|---|---|
| Characters returned | **16 760** | 73 477 | 29 533 |
| Approx. input tokens | **~4 190** | ~18 369 | ~7 383 |
| **Headings preserved** | **14** | 14 | **3** |

Two honest conclusions:

- Against a **browser-based crawl that keeps page furniture** (RAG Web Browser):
  **77% fewer tokens** for the same content sections.
- Against a **well-tuned HTML crawler** (Website Content Crawler): token counts are
  roughly equal — but that crawler flattens the document to **3 headings**, while this
  Actor keeps all **14**.

#### Why headings matter more than they look

RAG pipelines chunk documents. Headings are the natural chunk boundaries — they carry
the section topic into the embedding. A document flattened to 3 headings gives your
splitter almost nothing to work with, so chunks end up cut mid-argument and retrieval
quality drops.

Structure is not cosmetic here. It is what makes the extracted text usable.

#### What token savings mean on a real workload

Processing 100 000 pages per month, against a browser-based crawl:

| | This Actor | Browser-based crawl |
|---|---|---|
| Tokens sent to your LLM | **0.42 B** | 1.84 B |
| Cost at $1 / M input tokens | **~$420** | ~$1 840 |

**~$1 400/month saved** on token spend alone — before counting compute.

### What you get

Input:

```json
{
  "urls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }],
  "outputFormats": ["markdown"]
}
```

Output (one record per URL):

```json
{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "finalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "status": "ok",
  "httpStatus": 200,
  "title": "Retrieval-augmented generation - Wikipedia",
  "description": "",
  "language": "en",
  "author": "",
  "published": "",
  "siteName": "Wikipedia",
  "wordCount": 2383,
  "reductionRatio": 0.12,
  "markdown": "# Retrieval-augmented generation - Wikipedia\n\nFrom Wikipedia, the free encyclopedia\n\nType of information retrieval using LLMs\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources...\n\n## RAG and LLM limitations\n\n...",
  "links": []
}
```

Note how the Markdown opens directly with the title and first sentence — no navigation
preamble to pay for.

### Any language

Structural extraction only — no keyword or language assumptions. Verified on English,
Polish, German, Japanese and French Wikipedia in a single run:

| Language | Words extracted | Size vs original |
|---|---|---|
| English | 3 176 | 12.0% |
| Polish | 2 831 | 17.3% |
| German | 7 854 | 26.3% |
| Japanese | 5 578 | 16.6% |
| French | 18 124 | 26.6% |

The `language` field is returned per page, so you can route or filter downstream.

### How to use

1. Paste one or more URLs into **URLs**.
2. Leave **Output formats** as `markdown` (most token-efficient) — or add `text` / `html`.
3. Run. Each page becomes one dataset record.

Set **Max characters per page** to hard-cap your token budget, or turn off
**Keep links** to strip link markup and save even more.

#### From code

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("craftbox/fast-markdown-for-rag").call(run_input={
    "urls": [{"url": "https://example.com/article"}],
    "outputFormats": ["markdown"],
    "keepLinks": False,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["markdown"])
```

### Pricing

**$1.50 per 1 000 pages** processed. You are charged only for pages returned
with `status: ok` — failed and skipped URLs are free.

Compute on top of that is roughly **$0.21 per 1 000 pages**, because this Actor uses
plain HTTP requests instead of a headless browser.

Total: about **$1.71 per 1 000 pages** — and that is before the token savings above.

### Limitations — read this before you run

Stated plainly, so there are no surprises:

1. **JavaScript is not executed.** Pages rendered entirely client-side (many SPAs) will
   return little content. When that happens, the record is flagged with
   `requiresJavascript: true` and a warning — rather than quietly returning a thin result
   as a success. Check that field.
2. **No search.** You provide URLs; this Actor does not query search engines.
3. **HTTP errors are failures, not empty content.** A 404 or 500 returns
   `status: "failed"` with the status code, never a blank page marked as OK.
4. **Rate limiting is on by default** (1 second per host). Raise concurrency or lower the
   delay only where the source permits it.

### Politeness and compliance

This Actor fetches only publicly accessible pages. It does not log in, does not bypass
any protection mechanism, and enforces a per-host delay by default. You are responsible
for ensuring your use of the extracted content complies with the source's terms and with
applicable law.

### FAQ

**How does it compare to a headless-browser crawler?**
For static and server-rendered pages, this returns the same content sections with far
fewer tokens and roughly 10× lower compute cost. For client-rendered pages, a
browser-based crawler will extract more — use one of those instead, and see limitation 1.

**How does it compare to Website Content Crawler?**
Straight answer: on token count they are about equal, and that Actor is cheaper per page,
crawls whole sites and handles JavaScript. Pick this one when you fetch known URLs and
care about **document structure** — it keeps 14 headings where that crawler keeps 3,
which matters if you chunk by heading. Pick that one for whole-site crawls or JS-heavy pages.

**Will it break my Markdown structure?**
Headings, lists, tables, emphasis and links are preserved. Images are dropped by design,
as image markup is pure token cost in a RAG context.

**Can I get plain text instead?**
Yes — add `text` to `outputFormats`. You can request several formats in one run.

**How do I keep output small for a tight context window?**
Set `maxChars` to your budget and `keepLinks` to `false`. Records that get cut are marked
with `truncated: true`.

**Does it handle redirects?**
Yes. The resolved address is returned in `finalUrl`.

**What if a source rate-limits me?**
Enable a proxy in the input, raise `perHostDelay`, or lower `maxConcurrency`.
Temporary failures (429, 503) are retried with exponential backoff.

# Actor input Schema

## `urls` (type: `array`):

Pages to convert to Markdown. Any language is supported.

## `outputFormats` (type: `array`):

Markdown is the most token-efficient choice for LLM context.

## `maxChars` (type: `integer`):

Hard cap to protect your LLM token budget. 0 means no limit.

## `keepLinks` (type: `boolean`):

Include links in Markdown and return a list of outbound URLs. Turn off to save tokens.

## `maxConcurrency` (type: `integer`):

Parallel requests across all hosts.

## `perHostDelay` (type: `integer`):

Minimum interval between requests to the same host. Please stay polite to source servers.

## `timeout` (type: `integer`):

How long to wait for a single page before giving up.

## `maxRetries` (type: `integer`):

Retry attempts for temporary failures such as HTTP 429 or 503.

## `proxyConfiguration` (type: `object`):

Optional. Use a proxy when a source rate-limits your requests.

## Actor input object example

```json
{
  "urls": [
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ],
  "outputFormats": [
    "markdown"
  ],
  "maxChars": 0,
  "keepLinks": true,
  "maxConcurrency": 10,
  "perHostDelay": 1,
  "timeout": 20,
  "maxRetries": 2,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("craftbox/fast-markdown-for-rag").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }] }

# Run the Actor and wait for it to finish
run = client.actor("craftbox/fast-markdown-for-rag").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ]
}' |
apify call craftbox/fast-markdown-for-rag --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=craftbox/fast-markdown-for-rag",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/f9GTqNsLOwgUsIfiY/builds/Dh4ZHmzNbG3gcwcs4/openapi.json