Skip to content
Chalk Labs / AI SEO
How-to guide

How to Allow AI Crawlers in robots.txt

By Rahil Jain, founder of Chalk LabsUpdated How we research

The short answer

robots.txt allows every crawler it does not forbid, so a site with no AI-specific rules already allows AI crawlers at the robots.txt level. To allow them on purpose, add a group naming each bot, such as OAI-SearchBot, GPTBot and PerplexityBot, followed by Allow: /. Check that no blanket Disallow or firewall rule blocks them, then wait up to 24 hours for the change to take effect.

  • robots.txt allows by default. You only need to edit it if a rule, plugin or firewall blocks AI bots.
  • Each AI company runs separate bots for search, training and user requests. Allow or block each one on purpose.
  • A bot with its own group ignores your wildcard group, so repeat any private paths in its group.
  • robots.txt is a request, not a lock. OpenAI and Perplexity say user-triggered fetchers may not follow it.

Do you need to change robots.txt to allow AI crawlers?

Usually not. robots.txt is a small text file at the root of your site that tells crawlers which paths they may fetch. Under the Robots Exclusion Protocol, if no rule matches, access is allowed 1. So a site with no robots.txt, or one that blocks only private folders, already welcomes AI crawlers.

You need to edit it in three cases: a rule blocks all bots, a plugin or host added AI blocks you did not choose, or you want to write down your choices clearly. Cloudflare announced on 1 July 2025 that it was changing its default to block AI crawlers unless they pay creators 2. The post does not say whether that covers existing sites, so check your own settings in your CDN dashboard as well as robots.txt.

Before editing, look at your current file at yourdomain.com/robots.txt. The AI crawler checker reads it for you and flags blocks.

Do it step by step

0 of 7 done0%

  1. Open yourdomain.com/robots.txt in a browser. Confirm it loads with no error. Look for Disallow: / under User-agent: * and for any AI bot names already listed.

    Tip: If the file is missing, all bots are allowed. Create one only if you want to state your choices.

  2. For each company, choose allow or block for its search bot, its training bot and its user-request bot. Write the decision down in a small table before touching the file.

  3. Start from the templates below. Replace example paths with your own private folders. Repeat your private paths inside every named group.

    Tip: Keep comments in the file, with the date and the reason for each choice.

  4. The file must be named robots.txt, saved as plain text, and served at the top level of the domain. Each subdomain needs its own file. Reload the live URL and confirm the new text appears.

  5. In your CDN or security tool, look for bot-blocking and AI-crawler settings. Turn off any rule that blocks the search bots you chose to allow. Check that responses to bot visits are 200, not 403.

  6. Run the Python test below against your live file. It reports whether each bot name may fetch your page. Then also run the free crawler checker as a second opinion.

  7. Wait 24 hours. Then search your server logs for the bot names. You should see requests and 200 responses. If you see none after a week, that is not proof of a block, because bots visit on their own schedule.

Your ticks are saved in this browser only.

Copy-ready templates

Allow AI search bots, block AI training bots

text
# AI search bots: allowed (user-triggered fetchers such as ChatGPT-User are left out on purpose)
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /private/

# AI training bots: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else
User-agent: *
Disallow: /private/

Replace /private/ with your own paths. ChatGPT-User, Perplexity-User and Claude-User are user-triggered fetchers; leave them out unless you have a reason.

Allow all the AI bots listed above

text
User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Disallow: /private/

User-agent: *
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Use only if you are happy for training use too. The Disallow line is repeated inside the named group on purpose.

Test which bots may fetch a URL (Python 3)

bash
python3 - <<'EOF'
from urllib.robotparser import RobotFileParser
site = "https://example.com"
page = site + "/services"
rp = RobotFileParser(site + "/robots.txt")
rp.read()
for bot in ["OAI-SearchBot", "GPTBot", "PerplexityBot", "ClaudeBot", "Claude-SearchBot", "Googlebot"]:
    print(bot, "->", "ALLOWED" if rp.can_fetch(bot, page) else "BLOCKED")
EOF

Python's parser is simple and does not apply the longest-match rule. Treat it as a quick check and confirm with your logs. A 403 on robots.txt makes Python report every bot as BLOCKED; check the file loads in a browser first.

Which AI crawlers should you allow?

Allow the search bots if you want to appear in AI answers. Decide on the training bots separately. The table lists the tokens each company documents. A token is the name you write on the User-agent line.

OpenAI says its settings are independent: OAI-SearchBot surfaces sites in ChatGPT search, GPTBot crawls content that may train its models, and ChatGPT-User handles user actions 3. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers 3. OpenAI recommends allowing OAI-SearchBot 3.

Documented AI crawler tokens (check each company's page before relying on a name)
TokenCompanyWhat it doesIf you block it
OAI-SearchBotOpenAISurfaces sites in ChatGPT searchNot shown in ChatGPT search answers 3
GPTBotOpenAICrawls content that may train modelsSignals no training use; search is a separate bot 3
ChatGPT-UserOpenAIVisits a page when a user asksRules may not apply to user-initiated visits 3
PerplexityBotPerplexitySurfaces and links sites in Perplexity resultsFewer or no results from your site 4
Perplexity-UserPerplexityFetches a page when a user asksGenerally ignores robots.txt 4
ClaudeBotAnthropicCollects content that may contribute to trainingOpts out of that collection 5
Claude-SearchBotAnthropicImproves search result qualityMay reduce visibility and accuracy in search results 5
Claude-UserAnthropicFetches pages when a user asks ClaudeMay reduce visibility in user-directed search 5
Google-ExtendedGoogleControls Gemini training and grounding useDoes not affect Google Search inclusion 6
Applebot-ExtendedAppleGoverns training use of Applebot dataDoes not crawl pages itself 7

How do you write the allow rules?

Name each bot on a User-agent line, then add Allow: /. Several User-agent lines can share one group, and the rules beneath apply to all of them. The protocol matches user-agent names without regard to case, but it matches path text with case 1.

One trap catches many sites. If a bot has its own group, it follows that group and not your wildcard group. So a private-folder rule under User-agent: * does not apply to GPTBot once GPTBot has its own group. Repeat the private paths in each named group.

The most specific rule wins. The protocol says the match with the most characters is used, and an Allow beats an equal-length Disallow 1. Keep the file small: Google reads up to 500 KiB of robots.txt 8.

How long do changes take, and what are the limits?

Allow up to 24 hours. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust 3. Perplexity says it may take up to 24 hours for its systems to reflect changes 4. Google generally caches robots.txt for up to 24 hours 8.

robots.txt also has limits. The protocol states that its rules are not a form of access authorization 1. OpenAI says ChatGPT-User visits are user-initiated, so robots.txt rules may not apply 3. Perplexity says its user fetcher generally ignores robots.txt for the same reason 4.

Compliance claims are also contested. In August 2025 Cloudflare alleged that Perplexity used undeclared crawlers to reach sites that had blocked it 9. Perplexity denied this and said Cloudflare confused its traffic with a third-party browser service's 10. We cannot settle that dispute. If content must be private, use a login, not robots.txt.

What mistakes block AI crawlers by accident?

The classic mistake is a leftover Disallow: / under User-agent: * from a staging site. Check for it first. A close second is a security plugin or CDN setting that blocks bots by user-agent and overrides robots.txt entirely, so the file looks fine while the bots get errors.

Other mistakes are typing a token wrong, such as GPT-Bot for GPTBot, saving the file with the wrong name or folder, and blocking script and style files that Apple's crawler needs to render pages 7. Test the live file, not the copy in your editor.

A further mistake is assuming allowed means visited. Allowing a bot only removes a barrier. Confirm real visits in logs; our guide to monitoring AI bot activity in server logs shows how.

When should you not allow AI crawlers?

Block or limit them for paid content, private data, and pages with licensing limits. Use real access control for anything sensitive. If your business model depends on people visiting the page directly, weigh the trade-off honestly.

If you only care about search visibility, you do not need to allow training bots. For wider checks, see how to check if ChatGPT and other AI bots can crawl your website and what AI crawlers are.

Frequently asked questions

How do I allow GPTBot and PerplexityBot in robots.txt?

Add a User-agent: GPTBot line and a User-agent: PerplexityBot line followed by Allow: /. If nothing blocks them, they are allowed already. Also check your CDN for bot blocking, and wait up to 24 hours for changes to take effect 34.

Is OAI-SearchBot the same as GPTBot?

No. OpenAI documents OAI-SearchBot as the bot that surfaces sites in ChatGPT search, and GPTBot as the bot for content that may train its models 3. Each is controlled by its own robots.txt setting.

Does blocking Google-Extended remove me from Google AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search or act as a ranking signal 6. It controls whether crawled content may be used for Gemini training and grounding in Gemini Apps.

Does robots.txt stop all AI bots?

No. The standard says its rules are not access authorization 1, and OpenAI and Perplexity say user-triggered fetchers may not follow robots.txt 34. Use a login for private content.

Will allowing AI crawlers get me cited?

It removes one barrier only, and we do not guarantee citations. Your pages still need to be useful, trusted and readable without scripts. The guide on server-side rendering covers the last point.

About the author

Rahil Jain

Rahil Jain is an AI SEO (answer engine and generative engine optimisation) consultant and co-founder of Chalk Labs.

LinkedIn · Method and credentials

Sources

  1. RFC 9309: Robots Exclusion Protocol · IETF (RFC Editor)Opened October 20, 2026. Supports: No matching rule means access is allowed; user-agent matching is case-insensitive and paths are case-sensitive; most specific match wins and Allow beats an equal Disallow; rules are not access authorization.
  2. Content Independence Day: no AI crawl without compensation · CloudflareOpened October 20, 2026. Supports: On 1 July 2025 Cloudflare said it was changing its default to block AI crawlers unless they pay creators; the post does not say if existing sites are covered.
  3. Overview of OpenAI Crawlers · OpenAIOpened October 20, 2026. Supports: OAI-SearchBot, GPTBot and ChatGPT-User roles; independent settings; opted-out sites are not shown in ChatGPT search answers; about 24 hours for changes; robots.txt rules may not apply to user-initiated visits.
  4. Perplexity Crawlers · PerplexityOpened October 20, 2026. Supports: PerplexityBot surfaces sites in results and is not used for foundation model training; Perplexity-User generally ignores robots.txt; up to 24 hours for changes.
  5. Does Anthropic crawl data from the web, and how can site owners block the crawler? · AnthropicOpened October 20, 2026. Supports: ClaudeBot, Claude-User and Claude-SearchBot roles; disabling Claude-User or Claude-SearchBot may reduce visibility in search results.
  6. Google's common crawlers · Google Search CentralOpened October 20, 2026. Supports: Google-Extended is a standalone product token for Gemini training and grounding; it does not affect Google Search inclusion or ranking; it has no separate user-agent string.
  7. About Applebot · AppleOpened October 20, 2026. Supports: Applebot-Extended does not crawl webpages and only governs training use; blocked scripts or styles may stop Applebot rendering content properly.
  8. How Google interprets the robots.txt specification · Google Search CentralOpened October 20, 2026. Supports: 500 KiB size limit; generally cached up to 24 hours; most 4xx treated as no robots.txt; 5xx stops crawling for the first 12 hours.
  9. Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives · CloudflareOpened October 20, 2026. Supports: Cloudflare alleged on 4 Aug 2025 that Perplexity used undeclared crawlers on sites that blocked it.
  10. Agents or bots? Making sense of AI on the open web · PerplexityOpened October 20, 2026. Supports: Perplexity disputes Cloudflare's claims and says Cloudflare confused its traffic with unrelated BrowserBase traffic (4 Aug 2025)
How this page was made. Built on Rahil Jain's AI SEO method and drafted with AI research assistance. Every statistic links to the page we read it on (opened October 20, 2026 or nearby). Published October 20, 2026; last updated October 20, 2026. Chalk Labs sells AI SEO services, so we have a commercial interest in this topic. Spotted an error? Tell us and we will fix it.