# Rag Knowledge Graph Builder (`cspnair/rag-knowledge-graph-builder`) Actor

Transform websites into RAG-ready datasets. Crawls pages, chunks content into semantic segments (500-1000 tokens), and generates hypothetical questions for each chunk. No API key needed with native mode. Output: pre-indexed JSON optimized for AI retrieval with 3x better accuracy than raw text.

- **URL**: https://apify.com/cspnair/rag-knowledge-graph-builder.md
- **Developed by:** [csp](https://apify.com/cspnair) (community)
- **Categories:** AI, Developer tools, Agents
- **Stats:** 130 total users, 1 monthly users, 100.0% runs succeeded, 29 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $0.01 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RAG-Ready Knowledge Graph Builder

🧠 **Transform any website into a semantic dataset optimized for Retrieval-Augmented Generation (RAG)**

### The Problem

Traditional web scrapers produce giant walls of text (like `llms-full.txt`). For large sites, this approach has critical limitations:

- Exceeds context windows of most LLMs
- Models get "lost in the middle" of long documents
- Raw text provides no semantic structure for retrieval
- Poor retrieval accuracy in RAG pipelines

### The Solution

This Actor creates a **pre-indexed semantic dataset** that AI agents can ingest instantly with high accuracy:

1. **Intelligent Crawling** - Crawls websites following same-domain links
2. **Semantic Chunking** - Uses recursive character splitting to create logical segments (500-1000 tokens)
3. **Hypothetical Question Generation** - For every chunk, generates potential user questions using LLM
4. **RAG-Ready Output** - Structured JSON where each object contains chunk text, source URL, and hypothetical questions

### Why It's Better

Instead of raw data, you get **pre-indexed data** that skyrockets retrieval accuracy:

```json
{
  "chunkId": "abc123_0",
  "chunkText": "Apify is a platform for web scraping and automation...",
  "sourceUrl": "https://docs.apify.com/platform",
  "hypotheticalQuestions": [
    "What is Apify used for?",
    "How does Apify help with web scraping?",
    "What automation capabilities does Apify provide?"
  ],
  "tokenCount": 487,
  "metadata": {
    "pageTitle": "Apify Platform Overview",
    "crawledAt": "2024-01-15T10:30:00Z"
  }
}
```

### Input Configuration

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `startUrls` | array | required | URLs to start crawling from |
| `maxCrawlPages` | integer | 50 | Maximum pages to crawl (0 = unlimited) |
| `maxCrawlDepth` | integer | 3 | Maximum link depth from start URLs |
| `chunkSize` | integer | 750 | Target chunk size in tokens |
| `chunkOverlap` | integer | 100 | Overlapping tokens between chunks |
| `questionsPerChunk` | integer | 3 | Hypothetical questions per chunk |
| `llmProvider` | string | "openai" | LLM provider (openai/anthropic) |
| `llmModel` | string | "gpt-4o-mini" | Model for question generation |
| `openaiApiKey` | string | - | OpenAI API key (required for OpenAI) |
| `anthropicApiKey` | string | - | Anthropic API key (required for Anthropic) |
| `excludeSelectors` | array | \[...] | CSS selectors to exclude |
| `urlPatterns` | array | \[] | URL patterns to include |
| `excludeUrlPatterns` | array | \[...] | URL patterns to exclude |

### Output

#### Dataset (per chunk)

```json
{
  "chunkId": "unique_chunk_identifier",
  "chunkIndex": 0,
  "chunkText": "The actual text content...",
  "tokenCount": 523,
  "sourceUrl": "https://example.com/page",
  "pageTitle": "Page Title",
  "pageDescription": "Meta description",
  "hypotheticalQuestions": [
    "Question 1?",
    "Question 2?",
    "Question 3?"
  ],
  "questionsCount": 3,
  "metadata": {
    "crawledAt": "2024-01-15T10:30:00Z",
    "chunkStart": 0,
    "chunkEnd": 2100,
    "totalChunksInPage": 5
  }
}
```

#### Key-Value Store

- **OUTPUT** - Processing summary with statistics
- **rag-dataset.json** - Complete dataset as single JSON file

### Use Cases

#### 1. Build a Documentation Chatbot

Crawl your docs site and create a knowledge base for a customer support bot.

#### 2. Create a Research Assistant

Index academic papers or research sites for semantic search.

#### 3. Power a Content Discovery Engine

Build a recommendation system based on semantic similarity.

#### 4. Train Custom Embeddings

Use the chunks and questions to fine-tune embedding models.

### LLM Cost Estimation

Using GPT-4o-mini (~$0.15/1M input tokens, ~$0.60/1M output tokens):

- 100 pages × 5 chunks/page × 3 questions = ~$0.10-0.20

Using Claude 3 Haiku (~$0.25/1M input tokens, ~$1.25/1M output tokens):

- 100 pages × 5 chunks/page × 3 questions = ~$0.15-0.30

### Integration Examples

#### With LangChain

```python
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings

## Load the RAG dataset
chunks = load_apify_dataset("your-run-id")

## Create documents with questions as metadata
documents = []
for chunk in chunks:
    doc = Document(
        page_content=chunk["chunkText"],
        metadata={
            "source": chunk["sourceUrl"],
            "questions": chunk["hypotheticalQuestions"]
        }
    )
    documents.append(doc)

## Create vector store
vectorstore = Chroma.from_documents(documents, OpenAIEmbeddings())
```

#### With LlamaIndex

```python
from llama_index import Document, VectorStoreIndex

## Create documents from chunks
documents = [
    Document(
        text=chunk["chunkText"],
        metadata={
            "url": chunk["sourceUrl"],
            "questions": chunk["hypotheticalQuestions"]
        }
    )
    for chunk in chunks
]

## Build index
index = VectorStoreIndex.from_documents(documents)
```

### Technical Details

#### Chunking Strategy

- **Recursive Character Splitter** - Splits on semantic boundaries (paragraphs → sentences → words)
- **Token-based sizing** - Uses tiktoken for accurate GPT-4 token counting
- **Overlap handling** - Maintains context between chunks

#### Question Generation

- Uses system prompts optimized for retrieval-focused questions
- Generates diverse question types (what, how, why, when, etc.)
- Questions are self-contained and specific to chunk content

### License

ISC

### Support

For issues or feature requests, please open an issue on the repository.

# Actor input Schema

## `startUrls` (type: `array`):

List of URLs to start crawling from. The crawler will follow links within the same domain.

## `maxCrawlPages` (type: `integer`):

Maximum number of pages to crawl. Set to 0 for unlimited.

## `maxCrawlDepth` (type: `integer`):

Maximum depth of links to follow from start URLs.

## `chunkSize` (type: `integer`):

Target size for each content chunk in tokens. Recommended: 500-1000 for optimal RAG performance.

## `chunkOverlap` (type: `integer`):

Number of overlapping tokens between consecutive chunks to maintain context.

## `questionsPerChunk` (type: `integer`):

Number of hypothetical questions to generate for each chunk.

## `llmProvider` (type: `string`):

LLM provider for generating hypothetical questions. Use 'native' for free rule-based generation (no API key needed), or 'openai'/'anthropic' for higher quality AI-generated questions.

## `llmModel` (type: `string`):

Specific model to use (ignored for native provider). For OpenAI: gpt-4o-mini (cheap), gpt-4o (better). For Anthropic: claude-3-haiku-20240307 (cheap), claude-3-5-sonnet-20241022 (better).

## `openaiApiKey` (type: `string`):

Your OpenAI API key. Required if using OpenAI as LLM provider.

## `anthropicApiKey` (type: `string`):

Your Anthropic API key. Required if using Anthropic as LLM provider.

## `includeMetadata` (type: `boolean`):

Include page title, description, and other metadata in output.

## `excludeSelectors` (type: `array`):

CSS selectors for elements to exclude from content extraction (e.g., navigation, footer).

## `urlPatterns` (type: `array`):

Glob patterns for URLs to include. Leave empty to crawl all URLs on the domain.

## `excludeUrlPatterns` (type: `array`):

Glob patterns for URLs to exclude from crawling.

## `proxyConfiguration` (type: `object`):

Proxy settings for the crawler.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "maxCrawlPages": 50,
  "maxCrawlDepth": 3,
  "chunkSize": 750,
  "chunkOverlap": 100,
  "questionsPerChunk": 3,
  "llmProvider": "native",
  "llmModel": "gpt-4o-mini",
  "includeMetadata": true,
  "excludeSelectors": [
    "nav",
    "header",
    "footer",
    ".sidebar",
    ".navigation",
    ".menu",
    ".advertisement",
    ".ads",
    "#cookie-banner"
  ],
  "urlPatterns": [],
  "excludeUrlPatterns": [
    "**/*.pdf",
    "**/*.zip",
    "**/*.png",
    "**/*.jpg",
    "**/*.gif",
    "**/login*",
    "**/signup*",
    "**/auth*"
  ]
}
```

# Actor output Schema

## `summary` (type: `string`):

Summary of crawling and chunking results

## `chunks` (type: `string`):

All semantic chunks with hypothetical questions

## `overview` (type: `string`):

Quick overview of generated chunks

## `fullExport` (type: `string`):

Complete RAG-ready dataset as JSON

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("cspnair/rag-knowledge-graph-builder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("cspnair/rag-knowledge-graph-builder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ]
}' |
apify call cspnair/rag-knowledge-graph-builder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=cspnair/rag-knowledge-graph-builder",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/2K21joDTg3rofYgUL/builds/9y8QRvyCMFppFFcrF/openapi.json