# Site to Markdown — any site to clean, LLM-ready markdown (`topsail/site-to-markdown`) Actor

Scrape any website to clean, LLM-ready markdown — a compliant Firecrawl alternative for RAG ingestion, robots.txt always on.

- **URL**: https://apify.com/topsail/site-to-markdown.md
- **Developed by:** [Connor Teskey](https://apify.com/topsail) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Site to Markdown

**Turn any website into clean, LLM-ready markdown — one document per page, with robots.txt compliance locked on.**

Built for AI agents, RAG builders, and documentation pipelines that need a website-to-markdown step without running crawler infrastructure. Point it at a URL: it crawls breadth-first, strips navigation, ads, and boilerplate, and keeps only the main content as tidy markdown. If you have been looking for a Firecrawl alternative on Apify for scrape-to-markdown jobs, this is that actor.

### What you get

One dataset item per page:

| Field | Meaning |
| --- | --- |
| `url` | The URL that was requested. |
| `finalUrl` | URL after redirects. |
| `status` | HTTP status code (0 when the fetch itself failed). |
| `title` | Page title, when found. |
| `markdown` | Clean, LLM-ready markdown of the page's main content. |
| `text` | Plain-text version (only when `outputFormat` is `markdown+text`). |
| `linksCount` | Number of links discovered on the page. |
| `fetchedAt` | ISO-8601 fetch timestamp. |
| `rendered` | Whether a headless browser rendered the page (always `false` in v1). |
| `error` | Error message when the page failed, otherwise `null`. |

Every run also writes a `RUN_SUMMARY` record to the key-value store with page counts and a failure breakdown.

### Quick start

```json
{
    "startUrls": [{ "url": "https://docs.python.org/3/" }],
    "crawlMode": "site-crawl",
    "maxPages": 10,
    "maxDepth": 1
}
```

A run like this returns one markdown document per crawled page and typically finishes in well under a minute; the verification crawl of docs.python.org converted 5 of 5 pages.

### Output example

```json
{
    "url": "https://docs.python.org/3/tutorial/index.html",
    "finalUrl": "https://docs.python.org/3/tutorial/index.html",
    "status": 200,
    "title": "The Python Tutorial — Python 3.14.6 documentation",
    "markdown": "# The Python Tutorial\n\nPython is an easy to learn, powerful programming language. It has efficient high-level data st...",
    "linksCount": 35,
    "fetchedAt": "2026-06-11T00:49:18+00:00",
    "rendered": false,
    "error": null
}
```

### Why this one

- **Robots-locked by design.** Compliance is hard-coded into the crawler call, not an input default someone can flip. That makes the output safe to build a product on.
- **Selector-free extraction.** Main content is found by trafilatura with an automatic readability-style fallback — no CSS selectors to maintain when a site redesigns.
- **Honest zero-yield.** If no pages produce markdown, the run fails with a classified failure breakdown instead of finishing green on an empty dataset.
- **Precise scope control.** Include/exclude glob patterns match against the full URL, exclude wins, and same-domain crawling is the default.
- **Open foundation.** Built on [trawl](https://github.com/connorteskey/trawl) (MIT), a clean-room crawler, with [trafilatura](https://trafilatura.readthedocs.io/) as the quality extraction engine — the exact wheel is vendored into the image.

### Compliance and reliability

Topsail actors are built compliance-first and ship with self-healing plumbing:

- **robots.txt is always respected — locked on.** Every fetch goes through the crawler with robots compliance hard-coded; there is no input to turn it off. Pages disallowed by robots.txt are reported as `robots-blocked`, never fetched, and robots `Crawl-delay` is honored when larger than your politeness delay.
- **This actor reads only the public, static HTML pages you point it at** — the same documents any browser receives without logging in — and only where robots.txt permits.
- **Transient failures retry with backoff** (408, 425, 429, and 5xx responses, honoring `Retry-After`); persistent failures are reported, not hidden.
- **Every run writes a HEALTH summary** (`RUN_SUMMARY`) to the key-value store with page counts, a failure breakdown — `robots-blocked`, `http-4xx`, `http-5xx`, `timeout`, `extract-fail` — and a per-URL `failedPages` list, so you can see exactly which pages delivered and which were blocked, empty, or erroring. Only successful pages become dataset results.
- **No PII, no paywalled or login-gated content, no circumvention.**

### Pricing

Pay per result: $1.50 per 1,000 pages successfully extracted ($0.0015 per page), plus a fraction-of-a-cent actor start fee. Every dataset result is one extracted page — robots-blocked pages, failed fetches, and pages dropped by your URL filters never become results, so they cost nothing. The 10-page quick start above costs about two cents.

### Honest limits

- **No JavaScript rendering.** Static HTML only — SPAs that render entirely client-side will come back thin. Headless rendering is on the roadmap for v2.
- No sitemap.xml seeding yet; discovery is link-following from your start URLs.
- One markdown document per page; no site-level concatenated export (easy to build downstream from the dataset).
- robots.txt compliance cannot be disabled. If your use case requires ignoring robots.txt, this actor is not for you — by design.

### FAQ

**Is this a Firecrawl alternative?**
For the core scrape and crawl endpoints, yes: website to markdown, one clean document per page, ready for RAG ingestion — as an Apify actor instead of separate infrastructure. It does not replicate Firecrawl's JS rendering or search features in v1.

**Can it scrape JavaScript-heavy sites?**
Not in v1. It fetches static HTML, so server-rendered sites, documentation, and blogs work well; client-side SPAs come back thin.

**How do I scrape a single page to markdown?**
Set `crawlMode` to `single-page` and list your URLs in `startUrls`; each one is converted on its own with no link following.

**How do I keep a crawl focused on one section of a site?**
Use full-URL glob patterns: include `https://docs.example.com/en/*` and exclude `*/changelog/*`, for example. Exclude always wins.

**Can I turn off robots.txt compliance?**
No. It is hard-coded on, with no input to disable it. Disallowed pages are reported as `robots-blocked` so you can see what was skipped.

### More compliant data feeds from Topsail

- [GTA 6 Countdown & Developments Tracker](https://apify.com/topsail/gta6-tracker) — countdown, confirmed facts, diffed developments, market odds
- [Commodity Intel](https://apify.com/topsail/compliant-commodity-intel) — oil, gold, uranium headlines from permitted sources
- [Crypto News](https://apify.com/topsail/compliant-crypto-news) — BTC/ETH/DeFi headlines from major outlets
- [AI Research Radar](https://apify.com/topsail/compliant-ai-research-radar) — new papers and lab announcements

# Actor input Schema

## `startUrls` (type: `array`):

One or more URLs to convert. In single-page mode each URL is scraped on its own; in site-crawl mode each URL seeds a breadth-first crawl of the site.

## `crawlMode` (type: `string`):

Single-page scrapes only the start URLs; site-crawl follows links breadth-first up to Max pages and Max depth.

## `maxPages` (type: `integer`):

Stop crawling after this many pages per start URL (site-crawl mode). The prefill of 10 keeps the demo run around $0.02.

## `maxDepth` (type: `integer`):

How many link hops to follow from the start URL (site-crawl mode). Depth 1 means the start page plus the pages it links to.

## `includePatterns` (type: `array`):

Keep only pages whose full URL matches at least one glob, e.g. https://docs.example.com/en/\*. Leave empty to keep everything.

## `excludePatterns` (type: `array`):

Drop pages whose full URL matches any of these globs, e.g. */changelog/*. Exclude wins over include.

## `sameDomainOnly` (type: `boolean`):

Only follow links on the start URL's domain (site-crawl mode). Turn off to let the crawl cross to other domains, still capped by Max pages and Max depth.

## `engine` (type: `string`):

Auto tries trafilatura (highest quality) and falls back to the built-in readability-style extractor; the other options force one engine.

## `outputFormat` (type: `string`):

Markdown emits one markdown field per page; markdown+text adds a plain-text field too.

## `delayMillis` (type: `integer`):

Minimum delay between requests to the same host, in milliseconds. robots.txt Crawl-delay is honored on top of this when larger.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/"
    }
  ],
  "crawlMode": "site-crawl",
  "maxPages": 10,
  "maxDepth": 1,
  "includePatterns": [],
  "excludePatterns": [],
  "sameDomainOnly": true,
  "engine": "auto",
  "outputFormat": "markdown",
  "delayMillis": 500
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.python.org/3/"
        }
    ],
    "crawlMode": "site-crawl",
    "maxPages": 10,
    "maxDepth": 1
};

// Run the Actor and wait for it to finish
const run = await client.actor("topsail/site-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.python.org/3/" }],
    "crawlMode": "site-crawl",
    "maxPages": 10,
    "maxDepth": 1,
}

# Run the Actor and wait for it to finish
run = client.actor("topsail/site-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/"
    }
  ],
  "crawlMode": "site-crawl",
  "maxPages": 10,
  "maxDepth": 1
}' |
apify call topsail/site-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=topsail/site-to-markdown",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/zPCGiWf2SMQTGD6xr/builds/KLtUQS5DnskKSkfa9/openapi.json