llms.txt Generator: Make Any Website AI-Readable

Generate a publishable llms.txt for any website, built entirely from that site's own pages, titles and descriptions. Nothing is invented: every line in the output file traces back to something the site already says about itself.

1. Why use llms.txt Generator

An AI-readiness audit that reports "no llms.txt" leaves you with the work. Writing the file by hand means listing your key pages, grouping them sensibly, summarising each one, and keeping the whole thing in the format assistants expect.

The tempting shortcut is to have a language model write it. That produces a file describing pages that may not exist, with summaries that sound right and are subtly wrong — the worst outcome, since the file's purpose is to tell assistants what is true about your site.

This Actor takes the other route. It discovers your real pages through your sitemap, reads the title and meta description each page already publishes, groups them into sections, and assembles a spec-shaped file. The result is a draft you review, not a fabrication you have to fact-check.

2. Key features

• Sitemap-first discovery, with a link-following fallback when no sitemap is available.
• No invented content. Titles and summaries come from each page's own <title> and meta description.
• Automatic sectioning. Pages are grouped by URL structure into ordered sections.
• Site-name de-duplication. Titles like "Pricing | Acme" are stripped to "Pricing", so the file reads cleanly.
• The finished file is returned in full, ready to save as /llms.txt.
• Per-site isolation. Each site is crawled with its own request queue, so one site's crawl cannot starve the next.
• Configurable depth, from a handful of pages up to 100 per site.
• Bulk generation across many sites in a single run.

3. Who it's for

For marketing and SEO. Publish an llms.txt so ChatGPT and other assistants describe your product from the pages you chose, rather than from whatever they happen to crawl.

For documentation teams. Point it at your docs domain, get a sectioned index of your real pages, and publish it so assistants answering user questions reach current pages instead of stale ones.

For agencies. Generate a first-draft file for every client site in one run, then review and publish — a task that is otherwise an hour of copy-paste per site.

For developers and AI agents. Regenerate after a site restructure and diff against the published file to see which sections changed, as one step in a release pipeline.

4. How to use it

1. Open the Actor and click Try for free.
2. Put one site root per line in the URLs field.
3. Optionally set Max pages per site — the default of 30 suits most marketing sites; documentation sites benefit from more.
4. Click Start.
5. Open the Output tab, copy the llmsTxt value, review it, and save it as /llms.txt at your domain root.

5. Input parameters

Field	Type	Required	Default	Max	Description
urls	array of strings	Yes	["https://books.toscrape.com","https://quotes.toscrape.com"]	—	Site roots to generate for, one per entry.
maxPagesPerSite	integer	No	30	100	How many pages of each site are read, discovered via sitemap.xml and falling back to link-following.

6. Output

{
  "url": "https://example.com",
  "origin": "https://example.com",
  "llmsTxt": "# Example\n\n> A short summary taken from the homepage meta description.\n\nThis file lists the key pages of example.com for AI assistants. It was generated from the site's own page titles and descriptions.\n\n## Docs\n\n- [Getting started](https://example.com/docs/start): Install and run your first job.\n\n## Pricing\n\n- [Pricing](https://example.com/pricing)\n",
  "pagesIncluded": 24,
  "sectionsUsed": ["Docs", "Pricing", "Blog"],
  "generatedAt": "2026-07-29T10:22:14.000Z"
}

The llmsTxt field holds the complete file as text. That is the deliverable — copy it verbatim to /llms.txt.

A site that could not be crawled returns url, origin, generatedAt and an error. The error field is absent from successful rows.

7. Output fields

Field	Meaning
url	The site root as submitted.
origin	The normalised scheme + host that was crawled.
llmsTxt	The generated file, complete and ready to publish.
pagesIncluded	How many pages made it into the file. Excludes the homepage, which becomes the header — unless it is the only page, in which case it is listed.
sectionsUsed	The section names that ended up with at least one page, in file order.
generatedAt	ISO timestamp of generation.
error	Present only when generation failed.

8. How it works

Each site's sitemap.xml is read first, since it is the site's own statement about which pages matter. If no sitemap is available, the crawler follows links from the homepage instead, bounded by maxPagesPerSite.

For every page, the <title> and meta description are captured — nothing else, and nothing generated. Page titles usually repeat the site name as a suffix, so that suffix is detected and stripped, turning "Pricing | Acme" into "Pricing".

Pages are grouped into sections derived from URL structure and emitted in a fixed section order, so two runs of the same site produce a comparably-ordered file. The homepage's own metadata becomes the file's title and summary line, with the summary truncated to a readable length. Each entry carries its page description, also truncated, where one exists.

Every site is crawled with its own dedicated request queue. This is deliberate: sharing the run's default queue causes the second site's crawler to see the first site's handled requests and stop immediately at the page limit — a failure mode that silently produces empty files for every site after the first.

9. API & MCP usage

cURL

curl -X POST "https://api.apify.com/v2/acts/apifmcpfactory~llms-txt-generator/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com"],"maxPagesPerSite":30}'

As an Apify MCP tool. This Actor is callable directly by AI agents such as Claude and Cursor. Usage bills through your own Apify account.

https://mcp.apify.com?tools=apifmcpfactory/llms-txt-generator

Claude Desktop (claude_desktop_config.json):

{
    "mcpServers": {
        "llms-txt-generator": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com?tools=apifmcpfactory/llms-txt-generator",
                "--header",
                "Authorization: Bearer YOUR_APIFY_TOKEN"
            ]
        }
    }
}

As an MCP tool. Generation is also available as a hosted MCP server for AI agents via the llms.txt Suite, as generate_llms_txt.

10. Pricing

$0.02 per site generated, billed per site written to the dataset. Sites that fail to crawl are not charged.

The higher unit price relative to auditing reflects the work: generation crawls up to 100 pages per site, where an audit fetches four files.

11. Limits & performance

Memory	256 MB
Pages per site	30 by default, 100 maximum
Crawl concurrency	10 pages in parallel within a site
Per-page timeout	30 seconds, with up to 2 retries
Request queue	One per site

12. Limitations

• Output quality follows your metadata. Pages with no meta description produce link entries with no summary. The file is only as descriptive as the site.
• This is a draft for review, not a publish-blind artefact. Sectioning is derived from URL structure and may not match how you think about your own site.
• Page cap is real. A site with 5,000 pages will be represented by at most 100 of them; the cap exists to keep runs bounded and cheap.
• No llms-full.txt. This generates the index file, not the full-content variant.
• JavaScript-rendered content is not executed. Titles and descriptions must be present in the served HTML.
• Public pages only. Nothing behind a login is crawled.
• Publishing is your step. The file is returned in the output; it is not uploaded anywhere.

13. FAQ

What is llms.txt and where does it go? A plain-text index of your key pages for AI assistants, published at your domain root as /llms.txt.

Does this use AI to write the file? No. Every title and summary comes from the page's own HTML. That is the point — an invented file is worse than none.

How many pages will it include? Up to maxPagesPerSite, default 30 and maximum 100.

What if my site has no sitemap? It falls back to following links from the homepage.

How do I publish the result? Copy the llmsTxt field verbatim and serve it at https://yourdomain.com/llms.txt.

Should I edit it before publishing? Yes, review it. Section names come from URL structure and you may want to rename or reorder them.

How do I know whether it worked? Run our llms.txt Checker afterwards — a published llms.txt is the largest single component of the AI-readiness score.

Can I regenerate after changing my site? Yes, and comparing sectionsUsed and pagesIncluded between runs is a quick way to see what changed structurally.