# Sitemap Generator — XML Sitemap Crawler (`junipr/sitemap-generator`) Actor

Crawl a website, discover indexable URLs, and generate XML sitemaps with crawl diagnostics, canonical checks, status codes, and exportable URL inventory.

- **URL**: https://apify.com/junipr/sitemap-generator.md
- **Developed by:** [junipr](https://apify.com/junipr) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 5 total users, 1 monthly users, 72.4% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.90 / 1,000 page crawleds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Generator — XML Sitemap Crawler

### Introduction

Sitemap Generator is a production-grade Apify actor that crawls a website, discovers indexable URLs, and generates XML sitemaps with crawl diagnostics, canonical checks, status codes, and exportable URL inventory. It discovers accessible pages through link following, respects configurable depth limits and URL patterns, and detects existing sitemaps from `robots.txt` and common paths like `/sitemap.xml`.

**Primary use cases:**

- SEO professionals auditing and generating sitemaps for client sites
- Web developers building sitemaps for sites without CMS-generated ones
- DevOps teams automating sitemap generation in CI/CD pipelines
- Content teams verifying all pages are indexed and discoverable
- Migration specialists mapping old site structure for redirects

**Key differentiators:** JavaScript rendering support via Playwright for SPAs, automatic existing sitemap detection and merging, depth-based priority estimation with inbound link boosting, lastmod detection from HTTP headers and meta tags, and auto-split at 50K URLs per sitemap protocol spec.

### Why Use This Actor

| Feature | Sitemap Generator | Sitemap Generator (Apify) | XML Sitemap Creator | Screaming Frog |
|---------|-------------------|--------------------------|--------------------|-----------------------|
| JS-rendered pages | Yes (Playwright) | No | No | Yes (desktop) |
| Existing sitemap detection | Yes (robots.txt + paths) | No | Partial | Yes |
| Priority estimation | Depth + link count | None | Static values | Heuristic |
| lastmod from headers | Yes (multi-source) | No | No | Yes |
| changefreq estimation | Yes (content heuristic) | No | No | No |
| Output: XML + JSON | Both | XML only | XML only | XML + CSV |
| Auto-split >50K URLs | Yes with index | No | No | Yes |
| Canonical URL handling | Full support | No | No | Yes |
| PPE pricing | $4.90/1K pages | Compute-based | Compute-based | License fee |
| Zero-config | Yes | Yes | Mostly | No |

This actor handles the most common pain points with existing sitemap generators: failure on JavaScript-rendered pages, no detection of existing sitemaps, poor deduplication of query-parameterized URLs, and lack of meaningful priority values.

### How to Use

#### Zero-Config Quick Start

Just provide a start URL and run. Everything else has sensible defaults:

```json
{
    "startUrl": "https://example.com"
}
```

The actor crawls one page by default for a fast, reliable first run, generates an XML sitemap, and stores it in the Key-Value Store under the `SITEMAP_XML` key. Increase `maxPages` and `maxDepth` for a broader crawl.

#### Step-by-Step

1. Go to the actor's page on Apify Console
2. Enter your website URL in the **Start URL** field
3. (Optional) Adjust max pages, depth, or enable Playwright for JS-heavy sites
4. Click **Start** to run the actor
5. When complete, download the XML sitemap from the **Key-Value Store** tab (`SITEMAP_XML`)
6. Upload the sitemap to Google Search Console or place it at your site's root

#### Common Configuration Recipes

**Quick Audit** — Default settings for a fast overview:

```json
{
    "startUrl": "https://example.com",
    "maxPages": 100,
    "crawlerType": "cheerio"
}
```

**Full Site Map** — Larger bounded crawl:

```json
{
    "startUrl": "https://example.com",
    "maxPages": 5000,
    "maxDepth": 10,
    "crawlerType": "cheerio"
}
```

**Blog Only** — Generate sitemap for just the blog section:

```json
{
    "startUrl": "https://example.com/blog",
    "includePatterns": ["/blog/*"],
    "stayWithinPath": true
}
```

**SPA Site** — JavaScript-rendered single page application:

```json
{
    "startUrl": "https://app.example.com",
    "crawlerType": "playwright",
    "maxConcurrency": 5
}
```

**Compare with Existing** — See what your existing sitemap is missing:

```json
{
    "startUrl": "https://example.com",
    "existingSitemapAction": "compare"
}
```

### Input Configuration

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `startUrl` | string | *required* | Root URL to start crawling |
| `maxPages` | integer | 1 | Max pages to crawl (1-5,000) |
| `maxDepth` | integer | 0 | Link-following depth (0 = start URL only) |
| `includePatterns` | string\[] | `[]` | Glob patterns for URLs to include |
| `excludePatterns` | string\[] | file extensions | Glob patterns for URLs to exclude |
| `crawlerType` | string | "cheerio" | Engine: "cheerio" (fast) or "playwright" (JS) |
| `includeLastmod` | boolean | true | Include last modified dates |
| `includeChangefreq` | boolean | true | Include change frequency |
| `includePriority` | boolean | true | Include calculated priority |
| `checkExistingSitemap` | boolean | false | Detect existing sitemaps |
| `existingSitemapAction` | string | "merge" | merge, replace, or compare |
| `respectRobotsTxt` | boolean | true | Honor robots.txt directives |
| `sitemapFormat` | string | "xml" | Output: xml, txt, or both |
| `splitAtCount` | integer | 50000 | Auto-split threshold |

See the **Input Schema** tab for the complete list of parameters with detailed descriptions.

### Output Format

#### XML Sitemap (Key-Value Store: `SITEMAP_XML`)

```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-01-15T10:30:00Z</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2025-12-01T08:00:00Z</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>
```

#### Dataset Item (one per page crawled)

```json
{
    "url": "https://example.com/about",
    "statusCode": 200,
    "depth": 1,
    "title": "About Us",
    "lastModified": "2025-12-01T08:00:00.000Z",
    "changefreq": "monthly",
    "priority": 0.8,
    "contentType": "text/html",
    "responseTimeMs": 245,
    "inSitemap": true,
    "excludeReason": null
}
```

#### Run Summary (Key-Value Store: `RUN_SUMMARY`)

```json
{
    "startUrl": "https://example.com",
    "totalPagesCrawled": 347,
    "totalPagesInSitemap": 312,
    "pagesExcluded": 35,
    "pagesSkippedByRobots": 8,
    "pagesFailed": 3,
    "duplicatesRemoved": 12,
    "existingSitemapFound": true,
    "existingSitemapUrls": 290,
    "durationMs": 45200,
    "sitemapSplitCount": 1
}
```

### Tips and Advanced Usage

#### Optimizing Crawl Speed

- Use `crawlerType: "cheerio"` for static sites — it is 5-10x faster than Playwright and uses far less memory
- Increase `maxConcurrency` for faster crawls on sites that handle high request rates
- Set `excludeQueryParams: true` (default) to avoid crawling the same page with different query strings

#### URL Pattern Filtering

- Use `includePatterns` to limit the sitemap to specific sections: `["/blog/*", "/products/*"]`
- Use `excludePatterns` to skip admin pages, API endpoints, or file downloads
- Patterns use glob syntax: `*` matches anything except `/`, `**` matches anything including `/`

#### Existing Sitemap Workflows

- **Merge**: Combines crawled URLs with the existing sitemap for a complete picture
- **Compare**: Generates a diff report showing URLs missing from your current sitemap and URLs in the sitemap that are no longer accessible
- **Replace**: Ignores the existing sitemap entirely and generates a fresh one from the crawl

#### Submitting to Search Engines

After generating your sitemap, download it from the Key-Value Store and either upload it to Google Search Console or place it at your site root. You can also ping search engines programmatically: `https://www.google.com/ping?sitemap=https://example.com/sitemap.xml`

### Pricing

This actor uses **Pay-Per-Event (PPE)** pricing with the `page-crawled` event at **$4.90 per 1,000 pages crawled** in the queued monetization update.

A billable event occurs when the actor successfully fetches a URL, processes it, and records the result. You are NOT charged for URLs blocked by robots.txt, duplicate URLs filtered before request, failed requests, or the initial robots.txt and existing sitemap fetches.

#### Cost Examples

| Scenario | Pages | Cost |
|----------|-------|------|
| Small blog (50 pages) | 50 | $0.25 |
| Business site (200 pages) | 200 | $0.98 |
| E-commerce (5,000 pages) | 5,000 | $24.50 |
| News site (50,000 pages) | 50,000 | $245.00 |

Platform usage pass-through is queued because deep crawls and JavaScript rendering can create high compute cost.

### FAQ

#### Does it handle JavaScript-rendered pages?

Yes. Set `crawlerType` to `"playwright"` to enable full browser rendering. This handles React, Next.js, Vue, Angular, and any other SPA framework. The Playwright mode uses a real Chromium browser to render pages before extracting links, so it discovers routes that only exist in client-side JavaScript.

#### How does it estimate priority values?

Priority is calculated from two factors: page depth (distance from the homepage) and inbound link count. The homepage always gets priority 1.0. Each additional level of depth reduces priority by 0.2, down to a minimum of 0.1. Pages with many inbound links from other pages on the site receive a boost of up to +0.2.

#### Can it detect my existing sitemap?

Yes. When `checkExistingSitemap` is enabled, the actor checks `robots.txt` for Sitemap directives and probes common paths like `/sitemap.xml` and `/sitemap_index.xml`. It supports sitemap index files and will recursively fetch all sub-sitemaps.

#### What happens if my site has more than 50,000 URLs?

The sitemap protocol limits each sitemap file to 50,000 URLs. When this limit is exceeded, the actor automatically splits the output into multiple sitemap files and generates a sitemap index file (`SITEMAP_INDEX_XML` in the Key-Value Store) that references all the individual sitemaps.

#### Does it respect robots.txt?

Yes. The `respectRobotsTxt` option is enabled by default. The actor parses robots.txt for Disallow directives and Crawl-delay values. Disallowed paths are skipped entirely (never requested), and crawl delay is honored by reducing concurrency.

#### Can I filter which pages are included?

Yes. Use `includePatterns` to specify glob patterns for URLs that should appear in the sitemap (e.g., `["/blog/*"]`). Use `excludePatterns` to exclude specific paths or file types. Common binary file extensions are excluded by default.

#### How often should I regenerate my sitemap?

For most sites, weekly or monthly regeneration is sufficient. For news sites or frequently updated content, consider daily runs. You can schedule the actor on Apify to run automatically at any interval.

#### What's a "page crawled" for pricing purposes?

A page crawled is any unique URL that the actor successfully fetches and receives a response from (HTTP 2xx or 3xx). Pages that fail to load, URLs blocked by robots.txt, and duplicates filtered before the request is made are not counted as billable events.

# Actor input Schema

## `startUrl` (type: `string`):

The root URL to start crawling from. Must include protocol (https://).

## `maxPages` (type: `integer`):

Maximum number of pages to crawl. Higher values take longer but produce a more complete sitemap.

## `maxDepth` (type: `integer`):

Maximum link-following depth from start URL. 0 means only the start URL. Higher values discover deeper pages.

## `includePatterns` (type: `array`):

URL patterns to include (glob syntax). Empty means include all URLs. Examples: "/blog/*", "*.html", "/products/\*".

## `excludePatterns` (type: `array`):

URL patterns to exclude (glob syntax). Applied after include patterns. Common file extensions are excluded by default.

## `excludeQueryParams` (type: `boolean`):

Treat URLs with different query parameters as the same page for deduplication.

## `respectCanonical` (type: `boolean`):

Follow <link rel="canonical"> tags. Index the canonical URL and skip duplicates.

## `stayWithinDomain` (type: `boolean`):

Only crawl pages on the same domain as the start URL.

## `stayWithinPath` (type: `boolean`):

Only crawl pages under the same path prefix as the start URL.

## `includeLastmod` (type: `boolean`):

Include <lastmod> in the sitemap. Detected from Last-Modified header, <meta> tags, or schema.org dateModified.

## `includeChangefreq` (type: `boolean`):

Include <changefreq> in the sitemap. Estimated from content type heuristics.

## `includePriority` (type: `boolean`):

Include <priority> in the sitemap. Calculated from page depth and inbound link count.

## `defaultChangefreq` (type: `string`):

Default changefreq value when not estimable from content type.

## `sitemapFormat` (type: `string`):

Output format: xml (standard XML sitemap), txt (plain URL list), or both.

## `splitAtCount` (type: `integer`):

Auto-split sitemap at this many URLs and generate a sitemap index. Per sitemap protocol spec, max is 50,000.

## `checkExistingSitemap` (type: `boolean`):

Check for existing sitemaps at robots.txt and common paths before crawling.

## `existingSitemapAction` (type: `string`):

What to do when an existing sitemap is found: merge (combine with crawled), replace (ignore existing), or compare (output diff).

## `sitemapPaths` (type: `array`):

Additional paths to check for existing sitemaps, besides robots.txt.

## `crawlerType` (type: `string`):

Crawler engine: cheerio (fast, static HTML) or playwright (JS rendering for SPAs).

## `maxConcurrency` (type: `integer`):

Maximum concurrent requests (Cheerio) or browser pages (Playwright). Higher values crawl faster but use more resources.

## `requestTimeout` (type: `integer`):

Timeout per page request in milliseconds.

## `maxRetries` (type: `integer`):

Retry failed page requests up to this many times.

## `respectRobotsTxt` (type: `boolean`):

Honor robots.txt crawl directives and delays. Recommended to leave enabled.

## `userAgent` (type: `string`):

Custom User-Agent string sent with all requests.

## `proxyConfiguration` (type: `object`):

Proxy settings. Defaults to Apify datacenter proxies.

## `httpHeaders` (type: `object`):

Custom HTTP headers for all requests.

## `cookies` (type: `array`):

Cookies to set for all requests. Each object needs name, value, and domain.

## Actor input object example

```json
{
  "startUrl": "https://crawlee.dev",
  "maxPages": 1,
  "maxDepth": 0,
  "includePatterns": [],
  "excludePatterns": [
    "*.pdf",
    "*.zip",
    "*.jpg",
    "*.png",
    "*.gif",
    "*.svg",
    "*.css",
    "*.js"
  ],
  "excludeQueryParams": true,
  "respectCanonical": true,
  "stayWithinDomain": true,
  "stayWithinPath": false,
  "includeLastmod": true,
  "includeChangefreq": true,
  "includePriority": true,
  "defaultChangefreq": "weekly",
  "sitemapFormat": "xml",
  "splitAtCount": 50000,
  "checkExistingSitemap": false,
  "existingSitemapAction": "merge",
  "sitemapPaths": [
    "/sitemap.xml",
    "/sitemap_index.xml",
    "/sitemap/"
  ],
  "crawlerType": "cheerio",
  "maxConcurrency": 1,
  "requestTimeout": 30000,
  "maxRetries": 2,
  "respectRobotsTxt": true,
  "userAgent": "JuniprSitemapBot/1.0",
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "httpHeaders": {}
}
```

# Actor output Schema

## `results` (type: `string`):

Discovered URLs with metadata including last modified date, change frequency, priority, and HTTP status.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://crawlee.dev",
    "maxPages": 1,
    "maxDepth": 0,
    "defaultChangefreq": "weekly",
    "sitemapFormat": "xml",
    "splitAtCount": 50000,
    "existingSitemapAction": "merge",
    "crawlerType": "cheerio",
    "maxConcurrency": 1,
    "requestTimeout": 30000,
    "maxRetries": 2,
    "userAgent": "JuniprSitemapBot/1.0",
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("junipr/sitemap-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrl": "https://crawlee.dev",
    "maxPages": 1,
    "maxDepth": 0,
    "defaultChangefreq": "weekly",
    "sitemapFormat": "xml",
    "splitAtCount": 50000,
    "existingSitemapAction": "merge",
    "crawlerType": "cheerio",
    "maxConcurrency": 1,
    "requestTimeout": 30000,
    "maxRetries": 2,
    "userAgent": "JuniprSitemapBot/1.0",
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("junipr/sitemap-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://crawlee.dev",
  "maxPages": 1,
  "maxDepth": 0,
  "defaultChangefreq": "weekly",
  "sitemapFormat": "xml",
  "splitAtCount": 50000,
  "existingSitemapAction": "merge",
  "crawlerType": "cheerio",
  "maxConcurrency": 1,
  "requestTimeout": 30000,
  "maxRetries": 2,
  "userAgent": "JuniprSitemapBot/1.0",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call junipr/sitemap-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=junipr/sitemap-generator",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/38LcNTiPQ1aOWsgGh/builds/M9TVP5ZEjZrVOtOdC/openapi.json