Website Content Crawler Lite

Crawl public website pages quickly over HTTP and export clean text, Markdown, HTML, metadata, and discovered links with bounded concurrency, retries, robots.txt handling, optional proxies, and a machine-readable run summary.

Use this actor when you need a lightweight CSV, JSON, Excel, or API export of website content for AI/RAG ingestion, SEO audits, content monitoring, lead research, documentation indexing, or automation workflows.

At a glance

• Input: Start URLs, crawl depth, page limit, domain scope, include/exclude globs, output format, robots.txt, retries, concurrency, safe size limits, timeout, and optional proxy.
• Output: Requested URL, loaded URL, title, meta description, H1, text, Markdown or HTML, links, status code, content type, depth, parent URL, and scrape status.
• Output formats: Choose markdown, text, or html.
• Best for: Website-to-Markdown exports, lightweight crawling, AI knowledge ingestion, SEO checks, and content monitoring.
• Robots-aware: Can respect robots.txt before crawling pages.
• Failure-aware: Retries transient network, rate-limit, server, and challenge responses, then records actionable error rows and RUN_SUMMARY.
• Memory-safe defaults: Caps response size, exported content, discovered links, and concurrency to reduce out-of-memory failures.

Ready-to-run examples

Use these saved Store examples as starting points. Open any example to prefill the Actor input, then adjust URLs, keywords, limits, or filters for your own run.

• Crawl MDN Fetch API Reference Pages
• Crawl Crawlee Docs as Markdown
• Crawl Apify Pricing Content
• Crawl Example Domain as Plain Text
• Crawl Apify Platform Docs as Markdown
• Crawl Apify Store pages with cleaned HTML output
• View all ready-to-run examples (10 examples)

What can it do?

• Crawl public website pages: Follow links from start URLs within your configured scope.
• Export clean content: Save readable text plus Markdown or HTML output.
• Collect page metadata: Capture title, meta description, H1, status code, content type, depth, and parent URL.
• Discover links: Save absolute links found on each crawled page.
• Use as a website content API: Run from the Apify UI, API, schedules, webhooks, or the official Apify MCP server.

Common workflows

• AI/RAG ingestion: Export page Markdown for embeddings or knowledge bases.
• SEO audits: Collect title, description, H1, status code, and internal link data.
• Content monitoring: Schedule runs to track page text or metadata changes.
• Documentation indexing: Crawl docs pages and save Markdown rows.
• Research pipelines: Feed clean website text into downstream extraction or classification tools.

Input example

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxPages": 25,
  "maxDepth": 1,
  "sameDomainOnly": true,
  "includeGlobs": [],
  "excludeGlobs": ["**/login**", "**/signup**"],
  "outputFormat": "markdown",
  "respectRobotsTxt": true,
  "requestTimeoutSecs": 20,
  "maxConcurrency": 3,
  "maxRequestRetries": 2
}

Output example

{
  "url": "https://example.com",
  "loadedUrl": "https://example.com/",
  "title": "Example Domain",
  "description": "Example description",
  "h1": "Example Domain",
  "text": "Example Domain This domain is for use in illustrative examples...",
  "markdown": "Example Domain\\n\\nThis domain is for use in illustrative examples...",
  "links": ["https://www.iana.org/domains/example"],
  "statusCode": 200,
  "contentType": "text/html",
  "depth": 0,
  "parentUrl": null,
  "fetchedAt": "2026-07-03T10:00:00.000Z"
}

Input configuration

Setting	JSON key	What to enter
Start URLs	startUrls	Public HTTP/HTTPS pages where crawling should start.
Maximum pages	maxPages	Maximum rows to save, including skipped rows.
Maximum link depth	maxDepth	Link levels to follow from each start URL; use 0 for start pages only.
Stay on same domain	sameDomainOnly	Keeps crawling inside the start URL's domain.
Include URL globs	includeGlobs	Optional glob patterns that URLs must match.
Exclude URL globs	excludeGlobs	Glob patterns to skip, such as login or signup URLs.
Main content format	outputFormat	markdown, text, or html.
Respect robots.txt	respectRobotsTxt	Checks robots.txt and skips disallowed URLs.
Request timeout	requestTimeoutSecs	Maximum seconds to wait for one page response.
Maximum concurrency	maxConcurrency	Parallel page fetches; lower it for fragile or unusually large sites.
Maximum request retries	maxRequestRetries	Bounded retries for transient errors, rate limits, server errors, and challenge pages.
Graceful run time	runTimeSecs	Stops new batches before platform timeout and preserves partial results.
Response/content/link safety caps	responseSizeBytes, contentChars, linksPerPage	Bounds unusually large or link-heavy pages.
Proxy	proxyConfiguration	Optional Apify or custom proxy configuration; direct access is the default.

Output fields

Field	Description
url, loadedUrl	Requested URL and final loaded URL after redirects.
title, description, h1	Page title, meta description, and first H1.
text, markdown, html	Extracted page content; Markdown or HTML appears according to outputFormat.
links	Absolute links discovered on the page.
statusCode, contentType	Response status and content type.
depth, parentUrl	Crawl depth and parent URL that discovered the page.
fetchedAt, error, skippedReason	Timestamp plus failure or skip details when applicable.

Pricing

This Actor uses pay-per-event pricing: one small start fee plus a charge only when a usable HTML page is successfully stored. HTTP errors, robots.txt skips, non-HTML responses, filtered URLs, empty pages, and failed requests are saved for diagnosis but are not charged as extracted pages.

Event	Free	Bronze	Silver	Gold	Platinum	Diamond
Run start (once)	$0.005	$0.005	$0.005	$0.005	$0.005	$0.005
Extracted page	$0.00019876	$0.00017284	$0.00013481	$0.00010370	$0.000069135	$0.000048395

The dataset can contain more rows than charged pages because diagnostic skip/error rows are free. See the run's charge details and RUN_SUMMARY for exact extracted and failed counts.

Apify may also charge platform usage for compute, storage, proxies, or data transfer outside this Actor pricing. Check the Actor run and the Apify Pricing tab for the exact cost shown to your account.

Tips for best results

• Start with maxDepth: 0 or 1 before allowing a broader crawl.
• Keep sameDomainOnly enabled unless you intentionally want to crawl external links.
• Use includeGlobs for docs, blog, or product sections.
• Use excludeGlobs to skip login, signup, cart, checkout, search, and account pages.
• Choose markdown for RAG and LLM workflows; choose text when you want smaller rows.
• Keep the default concurrency and size limits until a representative crawl succeeds; increase them deliberately for larger sites.
• Add a proxy only when the target rate-limits or blocks direct requests.

Limits and caveats

• This is a lightweight crawler, not a full browser renderer. Heavy JavaScript pages may return limited content.
• The actor crawls public pages only and should not be used for private or logged-in pages.
• Non-HTML content is skipped and recorded with a skip reason.
• Robots.txt rules can skip pages when respectRobotsTxt is enabled.
• Redirects outside the configured domain scope and URLs resolving to private/local networks are blocked for safety.
• Responses larger than responseSizeBytes are rejected; exported text/Markdown/HTML is truncated at contentChars.
• A run with zero usable pages fails clearly instead of appearing successful with no results. Partial crawls stop cleanly at runTimeSecs and expose counts in RUN_SUMMARY.

API usage

curl "https://api.apify.com/v2/acts/fetch_cat~website-content-crawler-lite/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://example.com" }],
    "maxPages": 25,
    "maxDepth": 1,
    "outputFormat": "markdown"
  }'

Node.js:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/website-content-crawler-lite').call({
  startUrls: [{ url: 'https://example.com' }],
  maxPages: 25,
  outputFormat: 'markdown',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('fetch_cat/website-content-crawler-lite').call(run_input={
    'startUrls': [{'url': 'https://example.com'}],
    'maxPages': 25,
    'outputFormat': 'markdown',
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

MCP and AI agents

You can run this actor through the official Apify MCP server at https://mcp.apify.com.

For a focused single-actor tool, use:

https://mcp.apify.com?tools=fetch_cat/website-content-crawler-lite

Claude Code CLI:

$claude mcp add --transport http apify-website-crawler "https://mcp.apify.com?tools=fetch_cat/website-content-crawler-lite"

Generic MCP client configuration:

{
  "mcpServers": {
    "apify-website-crawler": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=fetch_cat/website-content-crawler-lite"
    }
  }
}

Example prompts:

• “Crawl the public documentation at https://example.com/docs to depth 1 and return Markdown for at most 20 pages.”
• “Extract titles, descriptions, H1s, and internal links from these five public URLs and summarize any failed pages.”

Agent-friendly inputs include startUrls, crawl scope and format controls, bounded concurrency/retries, response/content/link safety limits, and optional proxy configuration. Every run also exposes RUN_SUMMARY in the default key-value store.

FAQ

Can this crawl a whole website?
It can crawl multiple pages, but it is intentionally lightweight. Start with small limits and increase carefully.

Does it render JavaScript?
No. It fetches public HTML and extracts readable content from it.

Can I use the output for RAG?
Yes. Use outputFormat: "markdown" or text and export the dataset to your ingestion pipeline.

Can I export to CSV or Excel?
Yes. Download the Apify dataset as CSV, Excel, JSON, XML, RSS, or through the Dataset API.

Related actors

• Website Screenshot Generator
• Bing Search Results Scraper
• Google Autocomplete Scraper
• Google News Scraper
• WHOIS RDAP Domain Lookup

Support

If a run fails, returns no data, or a field looks wrong, open an issue from the Actor page.

Please include the Apify run ID or run URL, input JSON, one example public URL, query, or input item, what you expected, and what the dataset returned. Small reproducible inputs make parsing or site-layout issues much faster to fix.

Privacy and data handling

This Actor runs with Apify limited permissions and only processes data needed for the documented run. It uses the inputs you provide and the public records needed to produce the documented dataset to produce the output dataset and sends requests to the website URLs you provide; results are stored in Apify run storage for your account. FetchCat does not use your inputs or outputs for advertising, does not use them for model training, and does not retain them outside the Apify run except for transient support debugging when you explicitly share run details. You are responsible for using the Actor lawfully, respecting the target site's terms, and avoiding unnecessary personal or sensitive data in inputs.