# Documentation Crawler for RAG (`liquid_bark/docs-crawler-for-rag`) Actor

Specialized crawler for developer documentation sites. Detects frameworks (Docusaurus, GitBook, ReadTheDocs, MkDocs, Sphinx), extracts clean content, and outputs semantically chunked Markdown optimized for RAG pipelines.

- **URL**: https://apify.com/liquid\_bark/docs-crawler-for-rag.md
- **Developed by:** [Izz](https://apify.com/liquid_bark) (community)
- **Categories:** AI, Developer tools
- **Stats:** 5 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### What is Documentation Crawler for RAG?

This Actor crawls **developer documentation websites** and converts them into **semantically chunked Markdown** — ready for RAG pipelines, vector databases, and LLM applications. It automatically detects whether a site is built with **Docusaurus**, **GitBook**, **ReadTheDocs**, **MkDocs**, or **Sphinx**, and uses framework-specific selectors to extract only the documentation content — no sidebars, navigation, footers, or version badges.

Generic crawlers dump entire pages into one text blob, including all the UI noise. That degrades your RAG retrieval quality. This Actor solves that by understanding how documentation frameworks structure their HTML, and extracting only what matters.

#### What can this Actor do?

- **Detect documentation frameworks automatically.** Analyzes meta tags, CSS classes, URL patterns, and HTML structure to identify the framework. Each framework gets its own content selectors — Sphinx uses `.body`, Docusaurus uses `.theme-doc-markdown`, MkDocs uses `.md-content__inner`, and so on.

- **Split content into semantic chunks.** Splits at heading boundaries (H1 → H2 → H3) instead of by character count. Each chunk carries its heading path as metadata — e.g., `"section": "API > Authentication > OAuth"`. Code blocks are never split across chunks.

- **Preserve documentation hierarchy.** Each chunk includes breadcrumbs (page position in the docs tree) and section paths (heading hierarchy within the page). Your RAG system can filter by topic or weight results by depth.

- **Use llms.txt when available.** Checks for `/llms-full.txt` before crawling. If found, uses it directly — one HTTP request instead of crawling hundreds of pages. About 52% of major docs sites support this.

- **Deduplicate with content hashes.** SHA-256 hash per chunk. Compare hashes between crawl runs to detect changes without diffing full text.

- **Work on any site.** Unknown frameworks fall back to generic selectors (`article`, `main`, `[role='main']`). The Actor is not limited to the five supported frameworks.

#### What data does this Actor extract?

| Field | Example | Description |
|-------|---------|-------------|
| `url` | `https://docs.python.org/3/library/json.html` | Source page URL |
| `title` | `json — JSON encoder and decoder` | Page title (cleaned) |
| `content` | `## Basic Usage\n\njson.dump(obj, fp)...` | Clean Markdown, one chunk per item |
| `breadcrumbs` | `["Python Standard Library", "Internet Data Handling"]` | Page position in the docs tree |
| `framework` | `sphinx` | Detected framework |
| `section` | `json > Basic Usage` | Heading path within the page |
| `contentHash` | `a9925077f5be3d02...` | SHA-256 for deduplication |
| `chunkIndex` / `totalChunks` | `2` / `22` | Chunk position within the page |

### How to crawl documentation for your RAG pipeline

1. Click **Try for free** to open the Actor in Apify Console.
2. Enter one or more documentation URLs in **Start URLs** — e.g., `https://docs.python.org/3/library/` or `https://react.dev/learn`.
3. Set **Max Pages** (default: 100). The Actor discovers pages via sitemap.xml and link-following, so you only need the root URL.
4. Choose **Output Format**: `json` for semantic chunks (best for RAG), `markdown` for full pages, or `jsonl` for streaming.
5. Click **Start**. The Actor detects the framework, crawls, chunks, and pushes results to the dataset.
6. Download from the **Dataset** tab as JSON, CSV, or Excel — or fetch via API.

#### Using the Apify API

```bash
curl -X POST "https://api.apify.com/v2/acts/liquid_bark~docs-crawler-for-rag/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://docs.python.org/3/" }],
    "maxPages": 50,
    "outputFormat": "json",
    "chunkSize": 1500
  }'
```

Also available via [Python SDK](https://docs.apify.com/sdk/python) and [JavaScript SDK](https://docs.apify.com/sdk/js).

#### Integrating with a vector database

The JSON output is designed for direct ingestion into Pinecone, Weaviate, Qdrant, Chroma, or pgvector:

1. Run the Actor to crawl a documentation site.
2. Fetch the dataset: `GET /v2/datasets/{datasetId}/items`.
3. For each item: embed `content`, use `contentHash` as dedup key, store `section` + `breadcrumbs` as metadata.
4. On subsequent crawls, compare hashes to upsert only changed chunks.

### How much does it cost to crawl documentation?

**$2.00 per 1,000 output items** (pay-per-event). One item = one chunk in JSON mode or one page in Markdown mode. A typical page produces 3–5 chunks.

| Scenario | Pages | Items | Event cost | Compute | Total |
|----------|------:|------:|----------:|---------:|------:|
| Single library (e.g., Python `json`) | 1 | ~22 | ~$0.04 | <$0.01 | ~$0.05 |
| Small docs site | 50 | ~200 | ~$0.40 | ~$0.03 | ~$0.43 |
| Medium docs site | 100 | ~400 | ~$0.80 | ~$0.05 | ~$0.85 |
| Large docs site | 500 | ~2,000 | ~$4.00 | ~$0.25 | ~$4.25 |
| Full page export (Markdown) | 100 | 100 | ~$0.20 | ~$0.05 | ~$0.25 |

Sites with `renderJs: true` use more compute (headless browser).

### Input

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `startUrls` | array | *required* | URLs to start crawling. Follows links within the same domain. Max 100 URLs. |
| `maxPages` | integer | 100 | Maximum pages to crawl. 0 = unlimited (capped at 50,000). |
| `outputFormat` | string | `json` | `json` (chunks + metadata), `markdown` (full pages), `jsonl` (streaming). |
| `chunkSize` | integer | 1500 | Target chunk size in characters (200–10,000). May exceed target to keep code blocks intact. |
| `framework` | string | `auto` | `auto`, `docusaurus`, `gitbook`, `readthedocs`, `mkdocs`, `sphinx`. Override if auto-detection fails. |
| `renderJs` | boolean | false | Headless browser for JS-heavy sites (GitBook, newer Docusaurus). Slower, more compute. |
| `useSitemap` | boolean | true | Discover pages via sitemap.xml. Disable if sitemap is broken. |

### Output

Real example from crawling [Python `json` docs](https://docs.python.org/3/library/json.html) — this single page produced 22 chunks:

```json
{
  "url": "https://docs.python.org/3/library/json.html",
  "title": "json — JSON encoder and decoder",
  "content": "## Basic Usage\n\njson.dump(*obj*, *fp*, ***, *skipkeys=False*, ...)\n\nSerialize *obj* as a JSON formatted stream to *fp*...",
  "metadata": {
    "breadcrumbs": [],
    "framework": "sphinx",
    "section": "json — JSON encoder and decoder > Basic Usage",
    "contentHash": "a9925077f5be3d02e8f1c4a7b6d8e9f0...",
    "chunkIndex": 2,
    "totalChunks": 22
  }
}
```

Each chunk knows its section path, framework, and position. Your RAG pipeline can use `section` to filter by topic and `contentHash` for incremental updates.

### Supported frameworks

| Framework | Detected via | Extracts from | Example sites |
|-----------|-------------|---------------|---------------|
| **Docusaurus** | Meta generator, `data-theme`, theme CSS classes | `.theme-doc-markdown` | React, Crawlee, Jest |
| **GitBook** | URL domain, `gitbook-root` class, OpenGraph | `.page-body`, `.markdown-section` | Startup docs (often needs `renderJs: true`) |
| **ReadTheDocs** | URL domain, `rst-content` class | `.rst-content` | Django, Flask, Python projects |
| **MkDocs** | Generator meta tag, `md-content`, `data-md-component` | `.md-content__inner` | Pydantic, FastAPI |
| **Sphinx** | Generator meta tag, `sphinxsidebar`, `_static/` links | `.body` | Python stdlib, Linux kernel |
| **Unknown** | — | `article`, `main`, `[role='main']` | Any site with semantic HTML |

For each framework, the Actor removes framework-specific noise (sidebars, TOC, pagination, edit links, version selectors) before extracting content. This is the key difference to generic crawlers: a Docusaurus sidebar and a Sphinx sidebar have completely different HTML — the Actor knows how to strip both.

If you know which framework your target site uses, you can set the `framework` parameter to skip auto-detection. This is useful for sites where detection signals are hidden behind client-side rendering.

### llms.txt support

Before crawling, the Actor checks if the site provides [llms.txt](https://llmstxt.org/) files:

1. **`llms-full.txt` found** — Uses it directly as content source. No crawling needed — one HTTP request instead of hundreds of pages. The content is chunked using the same semantic splitting as crawled pages.
2. **`llms.txt` found (no `llms-full.txt`)** — Logged but crawled normally. `llms.txt` is typically just an index.
3. **Neither found** — Crawls via sitemap + link-following as usual.

Sites with `llms-full.txt` include Astro, Vue, AWS, LangChain, Crawlee, and Svelte documentation.

### Use cases

**RAG pipelines.** Crawl documentation sites, ingest chunks into a vector database (Pinecone, Weaviate, Qdrant, Chroma, pgvector), and use retrieval-augmented generation to answer questions. The `section` metadata lets you filter by topic, `contentHash` makes incremental updates efficient.

**Coding assistant context.** Feed chunks into Cursor, GitHub Copilot, or Claude so they reference up-to-date documentation when generating code. The heading hierarchy helps the model understand where each piece fits.

**Documentation search.** Build semantic search over technical docs. `breadcrumbs` and `section` provide faceted filtering, clean Markdown ensures results without UI noise.

**Change monitoring.** Run on a schedule, compare `contentHash` between runs. Detect API changes, deprecations, or new features in libraries you depend on.

**Documentation migration.** Export from multiple sources into uniform Markdown for import into a knowledge base or wiki.

### Examples

#### Python docs (Sphinx) — semantic chunks

```json
{
  "startUrls": [{ "url": "https://docs.python.org/3/library/" }],
  "maxPages": 50,
  "outputFormat": "json"
}
```

Outputs ~200 chunks with section paths like `"json > Basic Usage"` and `"os > File and Directory Access"`.

#### GitBook site — with JavaScript rendering

```json
{
  "startUrls": [{ "url": "https://docs.example.gitbook.io" }],
  "renderJs": true,
  "framework": "gitbook"
}
```

#### Full page export — Markdown, no chunking

```json
{
  "startUrls": [{ "url": "https://react.dev/learn" }],
  "maxPages": 100,
  "outputFormat": "markdown"
}
```

#### Multiple sites in one run

```json
{
  "startUrls": [
    { "url": "https://docs.python.org/3/" },
    { "url": "https://docs.pydantic.dev/latest/" }
  ],
  "maxPages": 200
}
```

Auto-detects Sphinx for Python and MkDocs for Pydantic. The `framework` field in each chunk tells you which site it came from.

### Tips for best results

- **Start with `auto` framework detection.** Only override if extraction looks wrong.
- **Use `renderJs: true` only when needed.** Try without it first — it is slower and costs more.
- **Set `maxPages` low initially** (10–20) to verify output quality before full crawls.
- **`chunkSize: 1500`** is a good default for RAG. Use 500–800 for precise Q\&A, 3000–5000 for summarization.

### FAQ

#### Is it legal to crawl documentation sites?

This Actor accesses only publicly available pages and respects `robots.txt` via Crawlee's built-in compliance. Developer documentation is published to be read and referenced. The Actor does not bypass authentication, paywalls, or access restrictions. Always check the target site's terms of service.

#### What if my site is not one of the supported frameworks?

The Actor still works. Unknown frameworks fall back to generic selectors (`article`, `main`, `[role='main']`) with standard noise removal. Output quality is usually good for any site with semantic HTML. Framework detection just makes extraction more precise.

#### Why are some chunks larger than the target size?

Code blocks are never split. If a code block exceeds the target, the entire block stays in one chunk. Headings are also hard boundaries — a short section becomes its own chunk rather than merging with the next one. Oversized paragraphs (large tables, lists, dense text) are split at structural boundaries — table rows, list items, or line breaks — while preserving table headers in each sub-chunk.

#### Does llms-full.txt content stay fresh?

The Actor uses `llms-full.txt` as-is without checking `Last-Modified` headers. If the file is stale, output may not reflect recent documentation changes. Disable the fast path by running without `useSitemap` and the Actor will crawl pages directly instead.

#### Can I run this on a schedule?

Yes. Use [Apify Schedules](https://docs.apify.com/platform/schedules) for daily/weekly crawls. Compare `contentHash` values between runs for automated documentation change detection.

#### How do I get help?

Open an issue on this Actor's Issues tab in Apify Console. Include the **Run ID** and the **URL** that caused the problem.

# Actor input Schema

## `startUrls` (type: `array`):

URLs to start crawling from. The crawler will follow links within the same domain.

## `maxPages` (type: `integer`):

Maximum number of pages to crawl. Set to 0 for unlimited (capped at 50,000).

## `outputFormat` (type: `string`):

Format of the output data.

## `chunkSize` (type: `integer`):

Target size for content chunks in characters. Only used with JSON output format.

## `framework` (type: `string`):

Documentation framework to optimize extraction for. Use 'auto' for automatic detection.

## `renderJs` (type: `boolean`):

Use a headless browser to render JavaScript. Enable for JS-heavy sites like GitBook or newer Docusaurus versions.

## `useSitemap` (type: `boolean`):

Try to discover pages via sitemap.xml before following links.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.example.com"
    }
  ],
  "maxPages": 100,
  "outputFormat": "json",
  "chunkSize": 1500,
  "framework": "auto",
  "renderJs": false,
  "useSitemap": true
}
```

# Actor output Schema

## `documentation` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.example.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("liquid_bark/docs-crawler-for-rag").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.example.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("liquid_bark/docs-crawler-for-rag").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.example.com"
    }
  ]
}' |
apify call liquid_bark/docs-crawler-for-rag --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=liquid_bark/docs-crawler-for-rag",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/0KCDWBRolytlHNcrG/builds/9V284FImZGIhBiAWp/openapi.json