llms.txt vs robots.txt vs sitemap.xml: What Each One Does
robots.txt tells crawlers where they may go. sitemap.xml lists the pages you want found. llms.txt is a proposed Markdown guide that points AI tools to your best pages. Set up robots.txt and a sitemap first. Google says Search does not use llms.txt, and a study of nearly 300,000 domains found no link between having one and AI citations 56.
- robots.txt controls crawler access, sitemap.xml lists URLs, and llms.txt is an optional, unofficial summary for AI tools.
- robots.txt is not security. The standard says its rules are not a form of access authorization.
- Google says Search does not use llms.txt, and files of that kind will neither harm nor help your Search visibility.
- Do robots.txt and a sitemap first. Add llms.txt only if it costs you little, and do not expect results from it.
What is the difference between llms.txt, robots.txt and sitemap.xml?
Each file answers a different question for a different reader. robots.txt answers: where may this crawler go? sitemap.xml answers: which pages exist, and when did they change? llms.txt answers: if an AI tool has little time, which pages should it read first?
Two of them are established. robots.txt is a Standards Track specification, RFC 9309 (a Proposed Standard) 1, and the sitemap format is a shared protocol documented at sitemaps.org 3. llms.txt is a proposal made by Jeremy Howard on 3 September 2024 4. No major search engine has committed to using it.
That difference in status drives the advice on this page. Use the two established files as the base of your technical setup, and treat llms.txt as an optional extra.
How do the three files compare side by side?
The table below sets the three files against the same questions. Each cell comes from the standard, the protocol page or the proposal itself.
| Question | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| What is it for? | Telling crawlers which parts of a site they may fetch 1. | Listing the URLs you want search engines to find 3. | Giving AI agents a short overview and links to detailed pages 4. |
| Format | Plain text, UTF-8, named /robots.txt at the top level 1. | XML. Required tags are urlset, url and loc 3. | Markdown. An H1 with the site name is the only required part 4. |
| Status | Standards Track, RFC 9309 (Proposed Standard) 1. | Shared protocol at sitemaps.org 3. | A proposal from September 2024 4. |
| Who reads it? | Crawlers that choose to obey. Google says reputable ones do 2. | Search engines. Bing uses lastmod as a key freshness signal 7. | Only tools that choose to. Google Search does not 5. |
| Can it hide a page? | No. Google says it is not a way to keep a page out of Google 2. | No. It only lists pages. | No. It only points to pages. |
| Does it promise visibility? | No. It permits access. | No. Submitting only shows the engine received the file 3. | No. Google says it neither helps nor harms Search visibility 5. |
| Size limits | Crawlers must parse at least 500 KiB; content past a crawler's limit may be ignored 1. | 50,000 URLs and 50 MB uncompressed per file 3. | None. Keep it short and curated. |
What does robots.txt do, and what can it not do?
robots.txt sets crawl rules for each named bot. Google says its main use is to avoid overloading your site with requests 2. For AI search, it decides whether bots such as OAI-SearchBot can fetch your pages. OpenAI says a site that blocks OAI-SearchBot will not be shown in ChatGPT search answers, and changes can take about 24 hours to apply 8.
It cannot protect private content. The standard says its rules are not a form of access authorization 1. Google adds that robots.txt is not a way to keep a page out of Google, and a blocked URL can still appear if other sites link to it 2. Use noindex or a login for that.
It also cannot bind every bot. Crawlers must choose to obey, so treat it as a polite request that major operators honour 2. For the AI bots to name, see GPTBot, OAI-SearchBot and ChatGPT-User and training bots versus search bots.
What does sitemap.xml do, and what can it not do?
A sitemap lists the pages you want found, so crawlers do not have to discover them by following links. Each file can hold up to 50,000 URLs and 50 MB uncompressed, and larger sites use a sitemap index 3. You can point crawlers to it with a Sitemap line in robots.txt 3.
The most useful field is lastmod, the date a page last changed. The protocol says it must be the page's change date, not the date the sitemap was built 3. Bing says lastmod is a key signal for how fast updates reach search results and AI answers, and that it ignores changefreq and priority 7.
A sitemap does not promise indexing. The protocol notes that a successful submission only shows the engine received the file 3. It also cannot fix thin pages. If a page is not worth citing, listing it changes nothing.
What does llms.txt do, and does it work?
llms.txt is a Markdown file at /llms.txt that gives an AI agent a short summary of your site and links to your most useful pages. The proposal says it is aimed mainly at inference, when an agent needs information to help a user, rather than at training 4. It has an H1 title, an optional summary and link lists.
The evidence for benefit is weak. Google's AI optimization guide says you do not need llms.txt for Search, that Google Search does not use it, and that creating one will neither harm nor help your visibility there 5. SE Ranking analysed nearly 300,000 domains in October 2026. Only 10.13% had a file, and it found no correlation with how often domains were cited by AI tools 6.
That does not prove no tool ever reads it. It means you should not count on it. Our llms.txt generator builds a clean file in minutes, and the how-to on creating llms.txt covers the steps.
What can none of these files do for AI visibility?
None of them makes an AI engine cite you. They help a machine reach and understand your pages. They do not make the pages worth citing.
What drives citations is the content itself: a direct answer, a clear source for each claim, and mentions elsewhere that confirm who you are. The query-level view is in what query fan-out is. Google also says no special schema.org markup or new machine-readable file is needed for generative AI search 5.
So think of the three files as plumbing. Good plumbing is necessary and invisible. It never replaces the water.
Which should you set up first, and how do they fit together?
Start with robots.txt, then the sitemap, then llms.txt if you want it. The order follows risk. A bad robots.txt can remove you from search results, while a missing llms.txt costs nothing proven.
The file below shows how robots.txt links to the sitemap and allows a search crawler while declining a training crawler. It is an illustration. Check each bot's current name in its owner's documentation before using it.
- User-agent: OAI-SearchBot
- Allow: /
- User-agent: GPTBot
- Disallow: /
- User-agent: *
- Allow: /
- Sitemap: https://www.example.com/sitemap.xml
What does a minimal llms.txt look like?
A minimal file follows the proposal's order: a title, a one-line summary, then sections of links with short notes 4. Keep it to the pages you would send a new customer to first.
- # Example Company
- > Example Company sells bookkeeping software to small shops in Pune.
- ## Key pages
- - [Pricing](https://www.example.com/pricing): plans and what each includes
- - [How it works](https://www.example.com/how-it-works): setup in three steps
- ## Optional
- - [Blog](https://www.example.com/blog): guides and updates
Frequently asked questions
Do I need llms.txt for AI search?
Does robots.txt keep my content private?
Will a sitemap get all my pages indexed?
No. The protocol says a successful submission only shows the engine received the file 3. A sitemap helps discovery. Whether a page is indexed or cited depends on the engine and the page itself.
Which bots should robots.txt allow for AI answers?
Allow the search crawlers of the engines you want to appear in. OpenAI says a site blocking OAI-SearchBot will not be shown in ChatGPT search answers 8. Training crawlers such as GPTBot are a separate choice, controlled independently.
Sources
- RFC 9309: Robots Exclusion Protocol · IETF (RFC Editor)Opened October 20, 2026. Supports: robots.txt purpose, location and UTF-8 text format; crawlers must parse at least 500 KiB; rules are not a form of access authorization.
- Introduction to robots.txt · Google Search CentralOpened October 20, 2026. Supports: Main use is managing crawler traffic; not a mechanism for keeping a page out of Google; crawlers must choose to obey; blocked URLs can still appear if linked.
- Sitemaps XML format · sitemaps.orgOpened October 20, 2026. Supports: 50,000 URL and 50 MB limits; required tags; lastmod must be the page's change date; Sitemap line in robots.txt; submission only shows receipt.
- The /llms.txt file · llmstxt.orgOpened October 20, 2026. Supports: Proposed by Jeremy Howard on 3 September 2024; Markdown format with H1 required; aimed mainly at inference.
- AI optimization guide · Google Search CentralOpened October 20, 2026. Supports: Google Search does not use llms.txt; creating one will neither harm nor help Search visibility; no special markup or AI files needed (page updated 2026-07-10).
- Does LLMs.txt impact your AI visibility and citations? · SE RankingOpened October 20, 2026. Supports: Nearly 300,000 domains; 10.13% had llms.txt; no correlation with AI citations; results specific to the model and dataset (15 October 2026).
- Keeping content discoverable with sitemaps in AI-powered search · Bing Webmaster Blog (Microsoft)Opened October 20, 2026. Supports: lastmod is a key freshness signal; changefreq and priority ignored; IndexNow recommended (31 July 2025).
- Overview of OpenAI Crawlers (Bots) · OpenAIOpened October 20, 2026. Supports: OAI-SearchBot surfaces sites in ChatGPT search; blocking excludes a site from answers; about 24 hours to take effect; independent of GPTBot.