# Fast Sitemap Generator (`eunit/sitemap-generator`) Actor

Boost SEO with this automatic Sitemap Generator. Crawl any site to create XML, HTML, & TXT sitemaps. Supports custom depth, regex filters, & robots.txt. Compatible with Google Search Console.

- **URL**: https://apify.com/eunit/sitemap-generator.md
- **Developed by:** [Emmanuel Uchenna](https://apify.com/eunit) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 34 total users, 3 monthly users, 100.0% runs succeeded, 4 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $80.00 / 1,000 successful crawling of a pages

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Generator

Automatically crawl any website and generate XML, HTML, and text sitemaps for SEO optimization. Perfect for submitting to Google Search Console, Bing Webmaster Tools, and improving search engine indexing.

### Features

- **Automatic Page Discovery**: Intelligently crawls websites following internal links.
- **Customizable Crawling**: Set crawling depth and apply filters to include/exclude specific pages.
- **Multiple Sitemap Formats**:
  - **XML**: Standard sitemap format for search engines.
  - **HTML**: Human-readable sitemap for visitors.
  - **Text**: Simple list of URLs.
- **Advanced Filtering**: Use Regex patterns to include or exclude specific URLs.
- **Robots.txt Support**: Automatically respects `robots.txt` rules (configurable).
- **Monetization**: Pay-Per-Event pricing model (charged per page successfully crawled).
- **Direct Connection**: This Actor uses a direct connection for crawling and does not support proxies. Note that for large-scale crawling, this might be limited by target site rate limits.

### Output

The Actor generates the following outputs:

#### Key-Value Store

1. **sitemap.xml**: Standard XML sitemap (if selected).
2. **sitemap.html**: HTML sitemap for users (if selected).
3. **sitemap.txt**: Plain text URL list (if selected).

#### Dataset

The dataset will contain the public URLs of the generated sitemap files. For example:

```json
{
  "format": "xml",
  "url": "https://api.apify.com/v2/key-value-stores/..../records/sitemap.xml"
}
```

### Input Configuration

#### Required

- **Start URLs** (`startUrls`): List of URLs to start crawling from.

#### Optional

- **Max Crawl Depth** (`maxCrawlDepth`): How deep to crawl (default: 3).
- **Max Pages Per Crawl** (`maxPagesPerCrawl`): Limit the number of pages (default: 30).
- **Include Patterns** (`includePatterns`): Regex patterns for URLs to include.
- **Exclude Patterns** (`excludePatterns`): Regex patterns for URLs to exclude.
- **Sitemap Formats** (`sitemapFormats`): "xml", "html", "txt" (default: "xml").
- **Respect robots.txt** (`respectRobotsTxt`): Whether to respect robots.txt (default: true).
- **Change Frequency** (`changefreq`): "always", "hourly", "daily", "weekly", "monthly", "yearly", "never" (default: "weekly").
- **Default Priority** (`defaultPriority`): Priority for pages, 0.0 to 1.0 (default: 0.5).
- **Include Images** (`includeImages`): Include image URLs in XML sitemap (default: false).

### Examples

#### Basic Usage

```json
{
  "startUrls": [ { "url": "https://example.com" } ],
  "maxCrawlDepth": 3,
  "sitemapFormats": ["xml"]
}
```

#### Advanced Usage

```json
{
  "startUrls": [ { "url": "https://example.com" } ],
  "maxCrawlDepth": 5,
  "maxPagesPerCrawl": 1000,
  "excludePatterns": [ ".*/admin/.*", ".*/login.*" ],
  "sitemapFormats": ["xml", "html"],
  "respectRobotsTxt": true,
  "changefreq": "daily",
  "priority": 0.8
}
```

### Related Actors

Expand your web scraping capabilities with these other powerful Apify Actors developed by the same author:

#### [Website Screenshotter](https://apify.com/eunit/website-screenshotter)

- **URL:** <https://apify.com/eunit/website-screenshotter>
- **Description:** Capture high-quality website screenshots. Features full-page capture, mobile emulation, custom resolutions, and export to JPG, PNG, or PDF. Fast and reliable. Export your result in any format of your choosing.

#### [LinkedIn Job Postings Scraper](https://apify.com/eunit/linkedin-job-postings-scraper)

- **URL:** <https://apify.com/eunit/linkedin-job-postings-scraper>
- **Description:** Designed for lead generation and career intelligence, this scraper extracts tailored **job titles, company names, locations, and job URLs** from LinkedIn. Essential for matching job market trends with real estate data (e.g., analyzing local economic activity).

#### [Target Product Reviews Scraper](https://apify.com/eunit/target-reviews-scraper)

- **URL:** <https://apify.com/eunit/target-reviews-scraper>
- **Description:** A tool for market research and e-commerce analysis. Scrape **product reviews, star ratings, and consumer sentiment** from Target.com, useful for businesses analyzing the retail landscape in target neighborhoods.

### Your feedback

We are always working on improving the performance of our Actors. So if you’ve got any technical feedback for the [Sitemap Generator](https://apify.com/eunit/sitemap-generator) Actor or simply found a bug, please create an issue on the [Actor’s Issues tab](https://console.apify.com/actors/Aclb73mv5h7pM5Vwq/issues) in [Apify Console](https://console.apify.com/actors/Aclb73mv5h7pM5Vwq).

# Actor input Schema

## `startUrls` (type: `array`):

List of URLs where the crawler will start. These should be the main pages of your website.

## `maxCrawlDepth` (type: `integer`):

Maximum depth of links to follow from start URLs. Depth 0 means only start URLs, depth 1 includes pages linked from start URLs, etc.

## `maxPagesPerCrawl` (type: `integer`):

Maximum number of pages to crawl. Use this to limit the scope of large websites.

## `includePatterns` (type: `array`):

Regular expression patterns for URLs to include. If empty, all URLs from the same domain are included.

## `excludePatterns` (type: `array`):

Regular expression patterns for URLs to exclude. Useful for filtering out admin pages, login pages, etc.

## `sitemapFormats` (type: `array`):

Select which sitemap formats to generate

## `respectRobotsTxt` (type: `boolean`):

Whether to respect the website's robots.txt file

## `changefreq` (type: `string`):

How frequently the page is likely to change. This is used in the XML sitemap.

## `defaultPriority` (type: `number`):

Default priority for pages (0.0 to 1.0). The homepage automatically gets 1.0, and priority decreases with depth.

## `includeImages` (type: `boolean`):

Include image URLs in the XML sitemap

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxCrawlDepth": 3,
  "maxPagesPerCrawl": 1000,
  "includePatterns": [],
  "excludePatterns": [
    ".*/admin/.*",
    ".*/login.*",
    ".*/wp-admin/.*"
  ],
  "sitemapFormats": [
    "xml"
  ],
  "respectRobotsTxt": true,
  "changefreq": "weekly",
  "defaultPriority": 0.5,
  "includeImages": false
}
```

# Actor output Schema

## `sitemapXml` (type: `string`):

No description

## `sitemapHtml` (type: `string`):

No description

## `sitemapTxt` (type: `string`):

No description

## `dataset` (type: `string`):

List of public URLs for the generated sitemaps

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ],
    "sitemapFormats": [
        "xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("eunit/sitemap-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev" }],
    "sitemapFormats": ["xml"],
}

# Run the Actor and wait for it to finish
run = client.actor("eunit/sitemap-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "sitemapFormats": [
    "xml"
  ]
}' |
apify call eunit/sitemap-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=eunit/sitemap-generator",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/Aclb73mv5h7pM5Vwq/builds/BVVJdLIAAZfry68fj/openapi.json