# RAG Post Processor - Text Cleaner & Chunker for LLM Pipelines (`jalicia/rag-post-processor`) Actor

Clean and chunk scraped text for RAG and LLM pipelines. Strips HTML, collapses whitespace, splits into overlapping chunks ready for embedding. Works standalone or chained after any scraper. Per-row billing.

- **URL**: https://apify.com/jalicia/rag-post-processor.md
- **Developed by:** [Jordan Wagner](https://apify.com/jalicia) (community)
- **Categories:** AI, Automation, Other
- **Stats:** 1 total users, 0 monthly users, 96.8% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.25 / 1,000 chunk processeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RAG Post Processor - Text Cleaner & Chunker for LLM Pipelines

Clean and chunk raw scraped text for RAG and LLM pipelines. Drop it after any scraper actor and get embedding-ready chunks in seconds.

### What it does

- Strips HTML tags and boilerplate
- Collapses whitespace and normalizes line breaks
- Splits text into overlapping chunks (default: 1000 chars, 100 overlap)
- Returns structured output with chunk index, length, and timestamp
- Works standalone or chained after Website Content Crawler and similar actors

### Input

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `data` | array | required | Array of objects from a previous scraper. Each object needs a `text`, `content`, `body`, or `html` field. |
| `chunk_size` | integer | 1000 | Max characters per chunk |
| `overlap` | integer | 100 | Character overlap between chunks |

#### Example input

```json
{
  "data": [
    { "text": "Your raw scraped content goes here. It can be long, messy HTML or plain text." }
  ],
  "chunk_size": 1000,
  "overlap": 100
}
```

### Output

Each chunk is returned as a dataset item:

```json
{
  "original_id": "item_0",
  "chunk_index": 0,
  "total_chunks": 3,
  "chunk_text": "Cleaned and chunked text ready for embedding...",
  "chunk_length_chars": 487,
  "cleaned_at": "2026-06-20 04:46:29.330000+00:00"
}
```

### Pricing

$0.0003 per output row. No subscription required — pay only for what you use.

### Use with PowerShell

Install the companion PowerShell module to call this actor from your automation scripts:

```powershell
Import-Module RAGPostProcessor
Invoke-RAGPostProcessor -InputText "Your scraped text here" -VerboseOutput
```

### Chaining with other actors

Works directly after **Website Content Crawler**, **Cheerio Scraper**, or any actor that outputs a `text` or `content` field. Use the Apify actor-to-actor API to pipe output from a scraper straight into this processor.

### Common use cases

- Preparing scraped web content for vector databases (Pinecone, Weaviate, Chroma)
- Cleaning LangChain / LlamaIndex document ingestion pipelines
- Pre-processing data for OpenAI embeddings or similar APIs
- Automating RAG pipeline data prep without custom code

# Actor input Schema

## `datasetId` (type: `string`):

Apify dataset ID from a previous actor run. Use this to chain directly after any scraper.

## `data` (type: `array`):

Array of objects from a previous scraper. Each needs a text, content, markdown, or html field.

## `chunk_size` (type: `integer`):

Maximum number of characters per chunk. Default is 1000.

## `overlap` (type: `integer`):

Number of characters to overlap between consecutive chunks. Default is 100. Capped at 50% of chunk\_size to bound the number of billed rows.

## `min_chunk_chars` (type: `integer`):

If the final chunk of an item is shorter than this, it's merged into the previous chunk instead of being emitted as its own (billable) row. No text is ever dropped. Default is 50.

## Actor input object example

```json
{
  "chunk_size": 1000,
  "overlap": 100,
  "min_chunk_chars": 50
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("jalicia/rag-post-processor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("jalicia/rag-post-processor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call jalicia/rag-post-processor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=jalicia/rag-post-processor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/rPRQKJP9bsGxsU9Ed/builds/smgifNubbbCXLsCHT/openapi.json