# Sitemap Extractor (`apify/sitemap-extractor`) Actor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

- **URL**: https://apify.com/apify/sitemap-extractor.md
- **Developed by:** [Apify](https://apify.com/apify) (Apify)
- **Categories:** Developer tools, SEO tools, Open source
- **Stats:** 244 total users, 51 monthly users, 81.0% runs succeeded, 5 bookmarks
- **User rating**: 3.12 out of 5 stars

## Pricing

from $0.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Extractor

This Actor is designed to bridge the gap between discovery and crawling. By traversing a website's `sitemap.xml` structure, it compiles a comprehensive list of all published pages and verifies their status before you commit resources to a full-scale scrape.

### Features

- **Recursive Sitemap Discovery:** Automatically detects and traverses nested sitemaps (sitemap indexes).
- **Efficiency:** Uses HTTP HEAD requests for URL validation, which are significantly faster and consume less bandwidth than full GET requests.
- **Proxy Support:** Integrated with Apify Proxy to prevent rate limiting or blocking during the discovery phase.
- **Detailed Output:** Provides the final URL, the corresponding HTTP status code, and the date-time of the page's last modification.

### How it Works

1. **Input:** You provide one or more "Start URLs" pointing to the domain name root, sitemaps or sitemap indexes.
2. **Extraction:** The Actor parses the XML, extracting both page URLs and links to further sitemaps.
3. **Validation:** For every page URL found, the Actor performs a status check (unless **Check page status** is disabled, see below).
4. **Deduplication:** The crawler uses unique keys to ensure that even if a URL appears in multiple sitemaps, it is only checked once.

### Listing sitemap URLs without status checks

By default (**Check page status** enabled), the Actor sends an HTTP HEAD request to every page found in the sitemaps and reports its status code. If you only need the list of URLs contained in the sitemaps and don't care about status codes, disable the **Check page status** option. In this mode the Actor:

- does **not** send any request to the individual pages. It only downloads and parses the sitemaps themselves;
- outputs each discovered URL with the `lastmod` value from the sitemap's `<lastmod>` tag, whenever the sitemap provides one. Since no pages are requested, the `Last-Modified` header fallback is not used, so URLs whose sitemap entry has no `<lastmod>` will have `lastmod: null`;
- omits the `status` field from the output.

This is dramatically faster and lighter on large sites, since it replaces one request per page with just a handful of requests for the sitemap files.

### Output

For each page URL, the Actor outputs:

| Field    |                                       Description                                       |
| :------- | :-------------------------------------------------------------------------------------: |
| `url`    |                                The page URL from the sitemap.                           |
| `status` | The HTTP status code returned by the HEAD request. Omitted when **Check page status** is disabled. |
| `lastmod` | Best-effort last-modification time (ISO 8601). See the note below.                      |

#### A note on last-modification data

The `lastmod` field is a single best-effort timestamp derived from two sources, in this order of preference:

1. The `<lastmod>` tag declared for the URL in the sitemap.
2. The `Last-Modified` HTTP header returned by the page (used only when the sitemap has no `<lastmod>`).

**We cannot guarantee that this information is available.** Both sources are optional: many sitemaps omit `<lastmod>` entirely, and a lot of servers don't send a `Last-Modified` header (this is especially common for dynamically generated pages). When neither source provides a value, `lastmod` is `null`. Even when present, the value is self-reported by the site and may not reflect the true last-modification time of the content.

When **Check page status** is disabled, no page requests are made, so only the first source (the sitemap's `<lastmod>` tag) is used.

### Usage

This Actor is ideal for:

- **Pre-crawling filter:** Generating a "clean" list of URLs for actors like *Website Content Crawler* or *Web Scraper*.
- **SEO Audits:** Quickly identifying 404 Not Found or 500 Server Error pages listed in your sitemap.
- **Site Mapping:** Getting a high-level overview of a site's architecture.

### Configuration

|          Field          |                           Description                           |
| :---------------------: | :-------------------------------------------------------------: |
|     **Start URLs**      | Just a domain name or a list of sitemap XML URLs to start from. |
| **Proxy configuration** |                   Settings for Apify Proxies.                   |

# Actor input Schema

## `startUrls` (type: `array`):

A static list of domains to scrape. <br><br>For details, see the <a href='https://apify.com/apify/sitemap-extractor#start-urls' target='_blank' rel='noopener'>Start URLs</a> section in the README.

## `checkPageStatus` (type: `boolean`):

When enabled (the default), the Actor sends a HEAD request to every page found in the sitemaps and outputs its HTTP status code. Disable this option to skip those requests entirely and simply output the list of page URLs (with <code>lastmod</code>) discovered in the sitemaps. This is much faster and lighter, but the output will not contain the <code>status</code> field, and <code>lastmod</code> falls back to the sitemap only (no <code>Last-Modified</code> header), so it will be <code>null</code> more often.

## `proxyConfiguration` (type: `object`):

Specifies proxy servers that will be used by the scraper in order to hide its origin.<br><br>For details, see <a href='https://apify.com/apify/sitemap-extractor#proxy-configuration' target='_blank' rel='noopener'>Proxy configuration</a> in README.

## `proxyRotation` (type: `string`):

This property indicates the strategy of proxy rotation and can only be used in conjunction with Apify Proxy. The recommended setting automatically picks the best proxies from your available pool and rotates them evenly, discarding proxies that become blocked or unresponsive. If this strategy does not work for you for any reason, you may configure the scraper to either use a new proxy for each request, or to use one proxy as long as possible, until the proxy fails. IMPORTANT: This setting will only use your available Apify Proxy pool, so if you don't have enough proxies for a given task, no rotation setting will produce satisfactory results.

## `maxCrawlingDepth` (type: `integer`):

Specifies how many sitemap levels the crawler will descend. This value is a safeguard against infinite crawling depths for misconfigured scrapers. <br><br>If set to <code>1</code>, will process only the root sitemap. If set to <code>0</code>, there is no limit.

## `maxRequestRetries` (type: `integer`):

The maximum number of times the scraper will retry to load each web page on error, in case of a page load error or an exception thrown by the <b>Page function</b>.<br><br>If set to <code>0</code>, the page will be considered failed right after the first error.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "checkPageStatus": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "proxyRotation": "RECOMMENDED",
  "maxCrawlingDepth": 0,
  "maxRequestRetries": 3
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items with discovered URLs, crawl status, and date-time of last modification.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("apify/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("apify/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call apify/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=apify/sitemap-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/rGeTNESChDZ65EbYh/builds/a0v1HRnZeRhkRWW9V/openapi.json