# AI Training Data Scraper (`blukaze/ai-training-data-scraper`) Actor

AI Training Data Scraper converts websites into clean, semantically-chunked, vector-ready data for LLMs, RAG pipelines, and AI search. Built for documentation, tutorials, and code-heavy content, with smart chunking and rich metadata.

- **URL**: https://apify.com/blukaze/ai-training-data-scraper.md
- **Developed by:** [Blukaze Automations](https://apify.com/blukaze) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 8 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 page processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Training Data Scraper

[![Apify Actor](https://img.shields.io/badge/Apify-Actor-blue)](https://apify.com)
[![Python 3.11](https://img.shields.io/badge/Python-3.11-green)](https://python.org)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Extract clean, semantically-chunked content from websites optimized for LLMs, RAG pipelines, and vector databases.**

Transform raw web content into AI-ready training data with intelligent chunking, rich metadata extraction, and vector database optimization.

***

### ✨ Key Features

- 🎯 **4 Smart Chunking Strategies**: Fixed token, sentence-based, semantic, and markdown section
- 🧹 **Intelligent Content Cleaning**: Removes navigation, ads, and boilerplate automatically
- 📊 **Rich Metadata Extraction**: Author, dates, keywords, language detection, content type
- 🔗 **Deep Recursive Crawling**: Crawl entire documentation sites with configurable depth
- ⚡ **Vector Database Ready**: Output formatted for Pinecone, Qdrant, Weaviate, ChromaDB
- 🦜 **LangChain & LlamaIndex Compatible**: Direct integration with popular AI frameworks
- 🎨 **Multiple Output Formats**: Markdown, plain text, JSON structured, vector-ready
- 🚀 **Dual Crawler Support**: Fast HTTP for static sites, Playwright for JS-heavy sites
- 🔒 **Respectful Crawling**: Respects robots.txt, configurable rate limiting

***

### 📋 Use Cases

| Use Case | Description |
|----------|-------------|
| **RAG Applications** | Build accurate retrieval-augmented generation systems with clean documentation |
| **AI Chatbots** | Train domain-specific chatbots on your knowledge base |
| **Code Assistants** | Extract technical documentation for programming assistants |
| **LLM Fine-tuning** | Collect high-quality training data for domain-specific models |
| **Semantic Search** | Populate vector databases for intelligent search systems |
| **Knowledge Management** | Structure and organize documentation for AI consumption |

***

### 🚀 Quick Start

#### 1. Basic Usage

```json
{
  "startUrls": [{"url": "https://docs.python.org/3/"}],
  "crawlerType": "cheerio",
  "maxCrawlPages": 100,
  "chunkingStrategy": "semantic",
  "outputFormat": "vector_ready"
}
```

#### 2. Advanced Configuration

```json
{
  "startUrls": [
    {"url": "https://docs.example.com/"},
    {"url": "https://api.example.com/docs"}
  ],
  "crawlerType": "playwright",
  "maxCrawlPages": 500,
  "maxCrawlDepth": 10,
  "chunkingStrategy": "semantic",
  "chunkSize": 512,
  "chunkOverlap": 100,
  "outputFormat": "vector_ready",
  "removeElements": ["nav", "header", "footer", ".sidebar", ".ads"],
  "includeMetadata": true,
  "proxyConfiguration": {"useApifyProxy": true}
}
```

***

### 📦 Chunking Strategies

Choose the right chunking strategy for your use case:

#### 1. Fixed Token (`fixed_token`)

- **Best for**: Consistent chunk sizes for embedding models with token limits
- **How it works**: Splits content into fixed-size token chunks (default: 512 tokens)
- **Use when**: You need precise control over chunk sizes for OpenAI/Anthropic embeddings

#### 2. Sentence-Based (`sentence_based`)

- **Best for**: Preserving natural language boundaries
- **How it works**: Groups complete sentences until reaching target size
- **Use when**: You want readable chunks that never cut mid-sentence

#### 3. Semantic (`semantic`) ⭐ Recommended

- **Best for**: Optimal RAG performance
- **How it works**: Uses NLP to detect topic boundaries and group related content
- **Use when**: Building RAG systems where context preservation is critical

#### 4. Markdown Section (`markdown_section`)

- **Best for**: Documentation and structured content
- **How it works**: Splits by heading hierarchy (## Section, ### Subsection)
- **Use when**: Scraping markdown-based documentation or wikis

***

### 📊 Output Schema

Each extracted page produces structured output:

```json
{
  "url": "https://docs.example.com/guide",
  "title": "Getting Started Guide",
  "content_format": "vector_ready",
  "full_content": "Complete page text...",
  "chunks": [
    {
      "id": "a1b2c3d4_chunk_0",
      "text": "Introduction to the framework...",
      "metadata": {
        "source_url": "https://docs.example.com/guide",
        "page_title": "Getting Started Guide",
        "chunk_index": 0,
        "token_count": 487,
        "has_code": true,
        "section_title": "Introduction",
        "language": "en",
        "content_type": "documentation"
      }
    }
  ],
  "metadata": {
    "author": "John Doe",
    "published_date": "2025-01-15T00:00:00Z",
    "language": "en",
    "keywords": ["python", "tutorial", "getting-started"],
    "word_count": 2500,
    "estimated_reading_time": 10,
    "content_type": "documentation",
    "has_code_blocks": true
  },
  "embedding_info": {
    "chunk_count": 8,
    "total_tokens": 3200,
    "ready_for_embedding": true,
    "recommended_model": "text-embedding-3-small"
  },
  "crawl_info": {
    "crawled_at": "2026-02-02T12:00:00Z",
    "crawl_depth": 2
  }
}
```

***

### 🔗 Integration Examples

#### LangChain

```python
from langchain.document_loaders import ApifyDatasetLoader
from langchain.schema import Document

def transform_dataset_item(item):
    documents = []
    for chunk in item.get("chunks", []):
        documents.append(Document(
            page_content=chunk["text"],
            metadata=chunk["metadata"]
        ))
    return documents

loader = ApifyDatasetLoader(
    dataset_id="your_dataset_id",
    dataset_mapping_function=transform_dataset_item
)

documents = loader.load()
```

#### LlamaIndex

```python
from llama_index import Document
from apify_client import ApifyClient

client = ApifyClient("your_api_token")
dataset = client.dataset("your_dataset_id").list_items().items

documents = []
for item in dataset:
    for chunk in item.get("chunks", []):
        documents.append(Document(
            text=chunk["text"],
            metadata=chunk["metadata"]
        ))
```

#### Pinecone

```python
import pinecone
from openai import OpenAI

## Initialize
pinecone.init(api_key="your-api-key")
index = pinecone.Index("your-index")
openai = OpenAI()

## Upsert chunks
for item in dataset:
    for chunk in item["chunks"]:
        # Generate embedding
        response = openai.embeddings.create(
            input=chunk["text"],
            model="text-embedding-3-small"
        )
        embedding = response.data[0].embedding
        
        # Upsert to Pinecone
        index.upsert([(
            chunk["id"],
            embedding,
            chunk["metadata"]
        )])
```

#### Qdrant

```python
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct

client = QdrantClient("localhost", port=6333)

points = []
for item in dataset:
    for chunk in item["chunks"]:
        embedding = embed_model.encode(chunk["text"])
        points.append(PointStruct(
            id=hash(chunk["id"]),
            vector=embedding.tolist(),
            payload=chunk["metadata"]
        ))

client.upsert(collection_name="docs", points=points)
```

***

### ⚙️ Configuration Reference

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `startUrls` | array | *required* | URLs to begin crawling |
| `crawlerType` | string | `"cheerio"` | `"cheerio"` (fast) or `"playwright"` (JS support) |
| `maxCrawlPages` | integer | `100` | Maximum pages to crawl |
| `maxCrawlDepth` | integer | `20` | Maximum crawl depth from start URLs |
| `chunkingStrategy` | string | `"semantic"` | Chunking algorithm to use |
| `chunkSize` | integer | `512` | Target chunk size (tokens/words) |
| `chunkOverlap` | integer | `100` | Overlap between chunks |
| `outputFormat` | string | `"vector_ready"` | Output format |
| `removeElements` | array | *see below* | CSS selectors to remove |
| `includeMetadata` | boolean | `true` | Extract page metadata |
| `extractLinks` | boolean | `false` | Include discovered hyperlinks in the output dataset (does not affect crawling) |
| `excludeUrlPatterns` | array | *see below* | URL patterns to skip |

#### Default Remove Elements

```json
["nav", "header", "footer", ".advertisement", "#cookie-banner", ".sidebar"]
```

#### Default Exclude Patterns

```json
["**/login**", "**/signup**", "**/register**", "**/cart**", "**/checkout**"]
```

***

### 💡 Pro Tips

#### For Best RAG Performance

- Use **semantic chunking** for intelligent topic grouping
- Set **chunk overlap to 15-20%** of chunk size (e.g., 100 for 512-token chunks)
- Enable **metadata extraction** for better filtering during retrieval
- Target **400-600 tokens** per chunk for most embedding models

#### For Large Documentation Sites

- Start with **maxCrawlPages = 50** to test configuration
- Use **Cheerio crawler** (10x faster) unless site requires JavaScript
- Set **exclude patterns** for login, user profiles, and dynamic pages
- Enable **Apify Proxy** to avoid rate limiting

#### For Code-Heavy Content

- **Markdown section chunking** preserves code block structure
- Extracted metadata includes **code languages detected**
- Code blocks are **never split** mid-block

***

### 🔧 Troubleshooting

#### Getting blocked by website

**Solution**: Enable Apify Proxy in configuration. For aggressive blocking, use residential proxies.

#### Missing content on JavaScript sites / Only 1 page crawled

**Solution**: Switch to `"crawlerType": "playwright"` for full JavaScript rendering. If your logs say "1 page crawled" and you are using Cheerio on a React/Vue/SPA app, the crawler is seeing an empty shell.

#### Chunks too large for embeddings

**Solution**: Reduce `chunkSize` to 384-512 tokens. Most models have 8192-token limits.

#### Empty chunks generated

**Solution**: Check `removeElements` - you may be removing content containers. Reduce selectors.

#### Slow crawling speed

**Solution**: Increase `maxConcurrency` (carefully) or use Cheerio crawler for static sites.

***

### 📈 Performance

| Metric | Cheerio Crawler | Playwright Crawler |
|--------|-----------------|-------------------|
| Speed | ~10 pages/sec | ~1 page/sec |
| JavaScript Support | ❌ No | ✅ Yes |
| Memory Usage | Low | Medium |
| Best For | Documentation, Blogs | SPAs, Dynamic Sites |

***

### 🏆 Why This Actor?

Compared to generic web scrapers, AI Training Data Scraper offers:

| Feature | Generic Scrapers | This Actor |
|---------|-----------------|------------|
| Token-Aware Chunking | ❌ | ✅ Uses tiktoken |
| Semantic Chunking | ❌ | ✅ NLP-based |
| Vector DB Ready | ❌ | ✅ Pre-formatted |
| Code Block Handling | ❌ | ✅ Never splits |
| Metadata Extraction | Basic | 15+ fields |
| RAG Optimization | ❌ | ✅ Purpose-built |

***

### 📞 Support

- **Issues**: Report bugs via Apify Console
- **Feature Requests**: Submit through Apify feedback
- **Documentation**: [Full API Reference](https://docs.apify.com)

***

### 📄 License

MIT License - See [LICENSE](LICENSE) file for details.

***

**Built for developers, optimized for AI.** ⚡

*Transform the web into training data.*

Created by [Blukaze Automation](https://blukaze.com)

# Actor input Schema

## `startUrls` (type: `array`):

List of URLs to begin crawling (documentation sites, forums, knowledge bases). The scraper will recursively crawl linked pages within the same domain.

## `crawlerType` (type: `string`):

Choose based on website complexity. Use 'cheerio' for static HTML sites (10x faster), 'playwright' for JavaScript-heavy sites that require rendering.

## `maxCrawlPages` (type: `integer`):

Maximum number of pages to process. Higher values will take longer but provide more comprehensive data.

## `maxCrawlDepth` (type: `integer`):

How many levels deep to crawl from start URLs. Depth 0 = only start URLs, Depth 1 = start URLs + their links, etc.

## `chunkingStrategy` (type: `string`):

How to split content into chunks for embedding. 'semantic' is recommended for best RAG performance.

## `chunkSize` (type: `integer`):

Target size in tokens. 512 is optimal for most embedding models (OpenAI, Anthropic).

## `chunkOverlap` (type: `integer`):

Number of tokens to overlap between chunks. Prevents context loss at chunk boundaries. Recommended: 15-20% of chunk size.

## `outputFormat` (type: `string`):

Format for the extracted content. 'vector\_ready' is optimized for direct ingestion into vector databases.

## `removeElements` (type: `array`):

CSS selectors for elements to remove from pages (navigation, ads, sidebars, etc.). These elements are stripped before content extraction.

## `includeMetadata` (type: `boolean`):

Extract and include rich metadata (author, date, keywords, language, content type). Useful for filtering during retrieval.

## `saveIndexPages` (type: `boolean`):

Whether to save navigation/index pages to the dataset. These are still crawled for links even if this is false. Default is true so you see results immediately, but can be set to false for pure content-only datasets.

## `extractLinks` (type: `boolean`):

Keep internal links within chunks for context. Can help with citation and source tracking.

## `saveScreenshots` (type: `boolean`):

Save page screenshots to key-value store (useful for debugging). Only works with Playwright crawler.

## `respectRobotsTxt` (type: `boolean`):

Whether to respect robots.txt rules. Recommended to keep enabled for ethical scraping.

## `maxConcurrency` (type: `integer`):

Maximum number of concurrent requests. Higher values are faster but may trigger rate limiting.

## `requestTimeout` (type: `integer`):

Maximum time to wait for a page to load before timing out.

## `proxyConfiguration` (type: `object`):

Proxy settings for crawling. Use Apify Proxy for reliable scraping.

## `urlPatterns` (type: `array`):

Glob patterns for URLs to include. Leave empty to crawl all URLs on the domain.

## `excludeUrlPatterns` (type: `array`):

Glob patterns for URLs to exclude (e.g., login pages, user profiles).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/"
    }
  ],
  "crawlerType": "cheerio",
  "maxCrawlPages": 1,
  "maxCrawlDepth": 0,
  "chunkingStrategy": "fixed_token",
  "chunkSize": 512,
  "chunkOverlap": 100,
  "outputFormat": "vector_ready",
  "removeElements": [
    "nav",
    "header",
    "footer",
    ".advertisement",
    "#cookie-banner",
    ".sidebar"
  ],
  "includeMetadata": false,
  "saveIndexPages": false,
  "extractLinks": false,
  "saveScreenshots": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 2,
  "requestTimeout": 30,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "urlPatterns": [],
  "excludeUrlPatterns": [
    "**/login**",
    "**/signup**",
    "**/register**",
    "**/cart**",
    "**/checkout**",
    "**/account**",
    "**/admin**"
  ]
}
```

# Actor output Schema

## `results` (type: `string`):

The main data extracted during the run.

## `debug_log` (type: `string`):

The full log of the Actor run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.python.org/3/"
        }
    ],
    "removeElements": [
        "nav",
        "header",
        "footer",
        ".advertisement",
        "#cookie-banner",
        ".sidebar"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("blukaze/ai-training-data-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.python.org/3/" }],
    "removeElements": [
        "nav",
        "header",
        "footer",
        ".advertisement",
        "#cookie-banner",
        ".sidebar",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("blukaze/ai-training-data-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/"
    }
  ],
  "removeElements": [
    "nav",
    "header",
    "footer",
    ".advertisement",
    "#cookie-banner",
    ".sidebar"
  ]
}' |
apify call blukaze/ai-training-data-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=blukaze/ai-training-data-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/of3dtkKW0R0x145qg/builds/KalrqFPo05jqP52i5/openapi.json