# Sitemap Generator (`datawinder/sitemap-generator`) Actor

Automatically crawl a website and generate an SEO-ready sitemap in XML, HTML, or TXT format. Supports crawl depth limits, URL include/exclude patterns, and optional merging with an existing sitemap.xml. Ideal for SEO audits, site migrations, and automation.

- **URL**: https://apify.com/datawinder/sitemap-generator.md
- **Developed by:** [DatawinderLabs](https://apify.com/datawinder) (community)
- **Categories:** Automation, SEO tools, Developer tools
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Generator

A powerful Apify Actor that automatically generates sitemaps for websites by crawling and discovering all accessible pages. Supports multiple output formats (XML, HTML, Text) and can merge with existing sitemaps.

### 🚀 Features

#### Automatic Page Discovery

- Intelligently crawls websites following internal links and navigation patterns
- Discovers all accessible pages automatically
- Only follows links from the same domain to prevent crawling external sites

#### Customizable Crawling

- **Crawl Depth Control**: Set maximum depth of crawling (0 = homepage only, 1 = homepage + direct links, etc.)
- **URL Filtering**: Include or exclude specific page types or directories using glob patterns
- **Request Limits**: Control the maximum number of pages to crawl

#### Multiple Sitemap Formats

- **XML**: Standard XML sitemap format compliant with Google and Bing specifications
- **HTML**: User-friendly HTML sitemap for website visitors
- **Text**: Plain text format, one URL per line

#### Sitemap Merging (XML Only)

- Fetch and merge with existing sitemap.xml files
- Preserves existing URLs while adding newly discovered ones
- New crawl data takes precedence over existing sitemap metadata

#### Built-in Validation

- Ensures sitemaps comply with Google and Bing specifications
- Proper priority settings based on page depth
- ISO 8601 date format for last-modified dates
- Validates XML structure and warns if exceeding 50,000 URLs (Google's limit)

### 📥 Input Parameters

#### Required

| Field | Type | Description |
|-------|------|-------------|
| `websiteUrl` | string | The URL of the website you want to generate a sitemap for. Example: `https://example.com` |

#### Optional

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `sitemapUrl` | string | - | URL to an existing sitemap.xml file. **Only used when format is XML.** If provided, the Actor will fetch and merge existing URLs with newly discovered ones. Ignored for HTML and Text formats. Example: `https://example.com/sitemap.xml` |
| `sitemapFormat` | string | `"xml"` | The file format for the generated sitemap. Options: `"xml"`, `"html"`, `"text"` |
| `maxCrawlDepth` | integer | `10` | Maximum depth of crawling. `0` = only start URL, `1` = start URL + all links from it, etc. Range: 0-50 |
| `includePatterns` | array | `[]` | Array of glob patterns for URLs to include. If empty, all URLs are included. Example: `["/blog/*", "/products/*"]` |
| `excludePatterns` | array | `[]` | Array of glob patterns for URLs to exclude. Example: `["/admin/*", "*.pdf", "/private/*"]` |
| `maxRequestsPerCrawl` | integer | `1000` | Maximum number of requests that can be made by this crawler. |

#### Example Input

```json
{
    "websiteUrl": "https://example.com",
    "sitemapFormat": "xml",
    "maxCrawlDepth": 3,
    "excludePatterns": ["/admin/*", "*.pdf"],
    "maxRequestsPerCrawl": 500
}
```

### 📤 Output Data

#### Key-Value Store

The Actor saves the generated sitemap file to the Key-Value Store:

- **XML Format**: `sitemap.xml` (Content-Type: `application/xml`)
- **HTML Format**: `sitemap.html` (Content-Type: `text/html`)
- **Text Format**: `sitemap.txt` (Content-Type: `text/plain`)

#### Dataset

The Actor also saves detailed metadata for each discovered URL to the Dataset:

| Field | Type | Description |
|-------|------|-------------|
| `url` | string | The URL of the page |
| `title` | string | The title of the page (extracted from `<title>` tag) |
| `lastModified` | string | ISO 8601 date when the page was last modified (crawl timestamp) |
| `priority` | string | Priority value for the sitemap (0.0 to 1.0). Calculated based on depth: homepage = 1.0, each level deeper decreases by 0.1 |
| `depth` | integer | Crawl depth of the page (0 = homepage, 1 = first level, etc.) |

#### Example Dataset Entry

```json
{
    "url": "https://example.com/about",
    "title": "About Us - Example.com",
    "lastModified": "2025-12-31T12:00:00.000Z",
    "priority": "0.9",
    "depth": 1
}
```

### 📋 Sitemap Format Details

#### XML Format

Standard XML sitemap compliant with [sitemaps.org](https://www.sitemaps.org/) protocol:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2025-12-31T12:00:00.000Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>1.0</priority>
  </url>
  <!-- More URLs... -->
</urlset>
```

**Features:**

- Valid XML structure with proper namespace
- Priority values (0.1 to 1.0) based on page depth
- ISO 8601 date format for last-modified dates
- Change frequency set to "weekly" for all URLs
- Validated against Google/Bing specifications

#### HTML Format

User-friendly HTML sitemap with clean styling:

```html
<!DOCTYPE html>
<html lang="en">
<head>
  <title>Sitemap</title>
  <!-- Styling included -->
</head>
<body>
  <h1>Sitemap</h1>
  <p>Total pages: 150</p>
  <ul>
    <li><a href="https://example.com/">Homepage</a></li>
    <!-- More links... -->
  </ul>
</body>
</html>
```

**Features:**

- Responsive design
- Clickable links with page titles
- Total page count displayed
- Clean, readable format

#### Text Format

Simple plain text format:

```
https://example.com/
https://example.com/about
https://example.com/contact
```

**Features:**

- One URL per line
- Simple and easy to parse
- Sorted by depth, then alphabetically

### 🔄 Sitemap Merging (XML Only)

When `sitemapFormat` is `"xml"` and `sitemapUrl` is provided:

1. The Actor crawls the website and discovers new URLs
2. Fetches the existing sitemap from the provided URL
3. Parses all URLs from the existing sitemap
4. Merges the URLs:
   - New URLs from crawl are added with fresh metadata
   - Existing URLs that are re-discovered keep the new crawl metadata (newer lastModified, updated priority)
   - Existing URLs that aren't re-discovered are preserved with their original metadata
5. Generates an updated sitemap with all URLs

**Note:** Sitemap merging only works with direct sitemap files (not sitemap index files). If a sitemap index is detected, a warning is logged.

### 💡 Use Cases

- **SEO Optimization**: Generate comprehensive sitemaps to improve search engine indexing
- **Website Maintenance**: Automatically update sitemaps when new pages are added
- **E-commerce Sites**: Create sitemaps for large product catalogs
- **Content Management**: Keep sitemaps synchronized with website content
- **Multi-format Support**: Generate different formats for different needs (XML for search engines, HTML for users)
- **Sitemap Updates**: Merge new discoveries with existing sitemaps without losing old URLs

### 🎯 Example Scenarios

#### Basic Sitemap Generation

```json
{
    "websiteUrl": "https://example.com",
    "sitemapFormat": "xml"
}
```

#### Generate HTML Sitemap with Limited Depth

```json
{
    "websiteUrl": "https://example.com",
    "sitemapFormat": "html",
    "maxCrawlDepth": 2,
    "maxRequestsPerCrawl": 100
}
```

#### Update Existing Sitemap

```json
{
    "websiteUrl": "https://example.com",
    "sitemapUrl": "https://example.com/sitemap.xml",
    "sitemapFormat": "xml",
    "maxCrawlDepth": 5
}
```

#### Exclude Specific Paths

```json
{
    "websiteUrl": "https://example.com",
    "sitemapFormat": "xml",
    "excludePatterns": ["/admin/*", "/private/*", "*.pdf", "*.zip"]
}
```

### ⚙️ Technical Details

- **Crawler**: Uses CheerioCrawler for fast HTML parsing (10x faster than browser-based crawlers)
- **Domain Filtering**: Automatically filters to only crawl links from the same domain
- **Priority Calculation**: Homepage (depth 0) = 1.0, each level deeper decreases by 0.1, minimum = 0.1
- **Validation**: Built-in validation ensures compliance with Google and Bing sitemap specifications
- **Performance**: Optimized for large websites with configurable request limits

### 📝 Notes

- The Actor only follows internal links (same domain) to prevent crawling external websites
- For very large websites, consider using `maxRequestsPerCrawl` to limit the crawl scope
- XML sitemaps with more than 50,000 URLs will generate a warning (Google's recommended limit)
- Sitemap merging is only available for XML format
- The Actor respects the crawl depth setting, so deeper pages may not be discovered if depth is too low

### 🔗 Resources

- [Sitemaps.org Protocol](https://www.sitemaps.org/)
- [Google Sitemap Guidelines](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview)
- [Apify Platform Documentation](https://docs.apify.com/)

# Actor input Schema

## `websiteUrl` (type: `string`):

The URL of the website you want to generate a sitemap for.

## `sitemapUrl` (type: `string`):

Optional URL to an existing sitemap.xml file. Only used when sitemap format is XML. If provided, the Actor will fetch and merge existing URLs with newly discovered ones. Ignored for HTML and Text formats.

## `sitemapFormat` (type: `string`):

The file format for the generated sitemap.

## `maxCrawlDepth` (type: `integer`):

Maximum depth of crawling. 0 means only the start URL, 1 means start URL + all links from it, etc.

## `includePatterns` (type: `array`):

Array of glob patterns for URLs to include. If empty, all URLs are included.

## `excludePatterns` (type: `array`):

Array of glob patterns for URLs to exclude (e.g., \['/admin/*', '*.pdf']).

## `maxRequestsPerCrawl` (type: `integer`):

Maximum number of requests that can be made by this crawler.

## Actor input object example

```json
{
  "websiteUrl": "https://example.com",
  "sitemapUrl": "https://example.com/sitemap.xml",
  "sitemapFormat": "xml",
  "maxCrawlDepth": 1,
  "includePatterns": [],
  "excludePatterns": [],
  "maxRequestsPerCrawl": 1000
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `keyValueStore` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("datawinder/sitemap-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("datawinder/sitemap-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call datawinder/sitemap-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=datawinder/sitemap-generator",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/KLleul5aQD5h4iOSD/builds/GcFyfSKR9wfSMCDNd/openapi.json