# AI Web Extractor: URL → Clean Markdown + JSON for LLM/RAG (`boxbox10/ai-web-extractor`) Actor

Turn any URL into clean, LLM-ready Markdown + structured JSON (title, headings, main content, links, metadata, token count). Perfect for RAG pipelines, AI agents, and LLM context.

- **URL**: https://apify.com/boxbox10/ai-web-extractor.md
- **Developed by:** [Marvin Eguilos](https://apify.com/boxbox10) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Web Extractor — URL → Clean Markdown + JSON for LLM & RAG

**Give it a URL. Get back clean, LLM-ready Markdown and structured JSON.** Title, headings, main content, links, metadata, and an accurate token count — every page, one tidy result. Built for RAG pipelines, AI agents, and anyone who needs the *content* of a page without the navbars, ads, cookie banners, and boilerplate.

> Feed the open web to your LLM the way it wants to be fed: as clean Markdown, with the token budget already counted.

***

### ✨ What it does

- **Main-content extraction** — Mozilla Readability strips nav, sidebars, ads, and footers so you keep just the article body.
- **HTML → Markdown** — high-fidelity conversion (GitHub-Flavored Markdown: tables, code blocks, lists, links) via Turndown.
- **Structured JSON** — `title`, `description`, `siteName`, `lang`, `byline`, `headings[]`, `links[]`, `wordCount`, `tokenCount`, `fetchedAt`.
- **Accurate token counts** — counted with the GPT/`cl100k`-family tokenizer so you know exactly how much context each page costs *before* you send it to a model.
- **Token budgeting** — optional `maxTokens` truncates output to fit your context window.
- **Robust by design** — one bad URL never kills the run. Failed pages return a clean error record (and are **never charged**).
- **Polite crawling** — respects `robots.txt` by default, sends a real User-Agent, and retries transient errors.
- **JS rendering when you need it** — flip `renderJs: true` to render client-side pages with a headless browser (opt-in, higher compute).

***

### 🎯 Use cases

| You want to… | This Actor gives you… |
|---|---|
| **Build a RAG knowledge base** | Clean Markdown chunks with token counts, ready to embed. |
| **Give an AI agent web context** | Structured JSON your agent can reason over — no HTML noise. |
| **Scrape docs / blogs to Markdown** | Publishable Markdown you can drop straight into a repo or wiki. |
| **Feed an LLM prompt** | Pre-counted tokens so you never blow the context window. |
| **Archive / snapshot pages** | Portable, diff-friendly Markdown + metadata. |

***

### 📥 Input

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | string\[] | — **(required)** | One or more URLs. Each successful page is one result. |
| `outputFormat` | `both` | `markdown` | `json` | `both` | Include Markdown, the JSON fields, or both. |
| `onlyMainContent` | boolean | `true` | Strip nav/ads/sidebars with Readability. |
| `includeLinks` | boolean | `true` | Include extracted absolute links + anchor text. |
| `maxPagesPerRun` | integer | `1000` | Safety cap on URLs processed per run. |
| `renderJs` | boolean | `false` | Render JS-heavy pages with a headless browser (higher cost). |
| `respectRobotsTxt` | boolean | `true` | Skip URLs disallowed by the site's `robots.txt`. |
| `maxTokens` | integer | `0` | Truncate Markdown to ~N tokens (`0` = no limit). |

#### Example input

```json
{
  "urls": [
    "https://example.com",
    "https://en.wikipedia.org/wiki/Markdown"
  ],
  "outputFormat": "both",
  "onlyMainContent": true,
  "includeLinks": true,
  "maxTokens": 0
}
```

***

### 📤 Output

One dataset item per URL. Successful example:

```json
{
  "url": "https://example.com/",
  "finalUrl": "https://example.com/",
  "statusCode": 200,
  "title": "Example Domain",
  "description": null,
  "siteName": null,
  "lang": "en",
  "byline": null,
  "excerpt": "This domain is for use in documentation examples...",
  "headings": [],
  "wordCount": 17,
  "tokenCount": 29,
  "fetchedAt": "2026-07-19T02:17:19.954Z",
  "markdown": "This domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
  "links": [
    { "url": "https://iana.org/domains/example", "text": "Learn more" }
  ]
}
```

Failed URL (returned, **not charged**):

```json
{
  "url": "https://not-a-real-domain-xyz.com/",
  "finalUrl": "https://not-a-real-domain-xyz.com/",
  "statusCode": null,
  "error": "getaddrinfo ENOTFOUND not-a-real-domain-xyz.com",
  "fetchedAt": "2026-07-19T02:17:19.831Z"
}
```

***

### 🔌 Call it from code (Apify API)

```bash
curl -X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~ai-web-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://en.wikipedia.org/wiki/Markdown"]}'
```

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const { defaultDatasetId } = await client
    .actor('YOUR_USERNAME/ai-web-extractor')
    .call({ urls: ['https://en.wikipedia.org/wiki/Markdown'] });
const { items } = await client.dataset(defaultDatasetId).listItems();
console.log(items[0].markdown);
```

***

### 💸 Pricing (Pay-Per-Event)

| Event | Price |
|---|---|
| Actor start | **$0.05** per run |
| Extracted page | **$0.003** per successful page |

- **You only pay for pages that succeed** — failed URLs are never charged.
- **🎁 Free tier:** free-plan users' platform usage is covered by Apify, so you can try it and run small jobs at **no cost** before scaling up.
- Cleaner structured JSON + accurate token counts than typical single-purpose "URL to Markdown" tools — at the same market price point.

***

### ⚖️ Acceptable use

This is a general-purpose format-conversion tool: **you supply the URLs and are responsible for having the right to crawl and use the content you submit.** By default the Actor respects `robots.txt` and identifies itself with a descriptive User-Agent. It does not target any single platform's private API and does not harvest personal data as a feature. Please crawl responsibly and comply with each site's terms of service and applicable law.

***

### 🧱 Under the hood

Node.js · [Crawlee](https://crawlee.dev) (Cheerio + optional Playwright) · [@mozilla/readability](https://github.com/mozilla/readability) · [Turndown](https://github.com/mixmark-io/turndown) (+ GFM) · [gpt-tokenizer](https://github.com/niieani/gpt-tokenizer) · Apify SDK. Stateless — nothing is stored between runs.

# Actor input Schema

## `urls` (type: `array`):

One or more URLs to extract. Each successful page is one billable result.

## `outputFormat` (type: `string`):

What to include per result: Markdown only, JSON fields only, or both.

## `onlyMainContent` (type: `boolean`):

Use Readability to strip nav, ads, sidebars, and footers — keep just the article body.

## `includeLinks` (type: `boolean`):

Include the list of extracted links (absolute URLs + anchor text) in each result.

## `maxPagesPerRun` (type: `integer`):

Safety cap on how many URLs are processed in a single run.

## `renderJs` (type: `boolean`):

Enable for JS-heavy / client-rendered pages. Uses a headless browser (higher compute cost). Leave off for most static pages.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed by the target site's robots.txt. You are responsible for having the right to crawl the URLs you submit.

## `maxTokens` (type: `integer`):

Optional. Truncate the Markdown output to approximately this many tokens (0 = no limit). Useful for fitting LLM context windows.

## Actor input object example

```json
{
  "urls": [
    "https://example.com"
  ],
  "outputFormat": "both",
  "onlyMainContent": true,
  "includeLinks": true,
  "maxPagesPerRun": 1000,
  "renderJs": false,
  "respectRobotsTxt": true,
  "maxTokens": 0
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com",
        "https://en.wikipedia.org/wiki/Web_scraping"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("boxbox10/ai-web-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://example.com",
        "https://en.wikipedia.org/wiki/Web_scraping",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("boxbox10/ai-web-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ]
}' |
apify call boxbox10/ai-web-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=boxbox10/ai-web-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/lifaDclcdZ2qfyOEv/builds/eUcL68IG6QfAlV25X/openapi.json