GPTBot vs OAI-SearchBot vs ChatGPT-User: What Is the Difference?
These are three separate OpenAI user agents with three jobs. OAI-SearchBot finds pages to show in ChatGPT search. GPTBot collects content that may train OpenAI's models. ChatGPT-User fetches a page when a person asks ChatGPT to. OpenAI treats each setting separately, so you can allow search and refuse training 1.
- Allow OAI-SearchBot if you want your pages to appear in ChatGPT search answers. OpenAI recommends this.
- Disallow GPTBot if you do not want your content used to train OpenAI's models. This does not remove you from ChatGPT search.
- ChatGPT-User acts for a person, so OpenAI says robots.txt rules may not apply to it.
- Changes to robots.txt can take about 24 hours to reach OpenAI's search systems.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
The difference is purpose. OpenAI runs each bot for one job and lets you set a robots.txt rule for each one 1. A user agent is the name a bot sends with every request so your server can tell who is visiting. robots.txt is the plain text file at the root of your site that tells bots which pages they may fetch.
OAI-SearchBot is the search bot. OpenAI says it surfaces websites in ChatGPT's search features, and that sites opted out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links 1.
GPTBot is the training bot. It crawls content that may be used to train OpenAI's generative AI foundation models 1. ChatGPT-User is the on-demand bot. OpenAI uses it for certain user actions in ChatGPT and Custom GPTs, and says it is not used to crawl the web automatically 1.
How do the three bots compare side by side?
The table below summarises what OpenAI documents for each bot. The wording in the last two columns is our reading of the OpenAI page 1.
| Question | OAI-SearchBot | GPTBot | ChatGPT-User |
|---|---|---|---|
| What is it for? | Showing sites in ChatGPT search | Collecting content that may train models | Fetching pages when a user asks |
| Runs automatically? | Yes, it crawls | Yes, it crawls | No, a person triggers it |
| What happens if you block it? | You are not shown in ChatGPT search answers | Your content should not be used for training | Unclear. OpenAI says robots.txt rules may not apply |
| Does it decide search inclusion? | Yes | No | No. OpenAI says it is not used for that |
| Does OpenAI recommend allowing it? | Yes | No recommendation stated | Not covered by a robots.txt rule |
| Does OpenAI publish IP ranges? | Yes, as a JSON file | Yes, as a JSON file | Yes, as a JSON file |
Should you allow OAI-SearchBot?
Yes, if you want ChatGPT to be able to show your pages. OpenAI says it recommends allowing OAI-SearchBot in your robots.txt file 1. Blocking it takes you out of ChatGPT search answers, which is the opposite of what most businesses want.
Allowing it is not a promise of a citation. It only makes your pages eligible. Whether ChatGPT then uses a page depends on the question, the competing pages and the quality of your answer. Our guide to getting recommended by ChatGPT covers that part.
One technical point matters here. Vercel's late-2024 study found that none of the major AI crawlers it tested, including OAI-SearchBot, rendered JavaScript 6. If your key text only appears after scripts run, the bot may see an empty page even when robots.txt allows it. See client-side vs server-side rendering for AI crawlers.
Should you block GPTBot?
That is a business choice, not a technical one. OpenAI says disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models 1. It does not say that blocking GPTBot removes you from ChatGPT search, because search is the job of OAI-SearchBot.
Reasons to block include paywalled or licensed content, original research you sell, and a general wish to keep content out of training sets. Reasons to allow include a belief that being in the training data helps models know your brand. We found no published evidence that allowing GPTBot raises citations, so treat that as an open question.
OpenAI adds a detail worth knowing: if you allow both OAI-SearchBot and GPTBot, it may use the results of one crawl for both purposes 1. If you want search without training, disallow GPTBot explicitly rather than hoping the two stay apart.
What is ChatGPT-User, and does robots.txt control it?
ChatGPT-User is the bot that visits a page because someone asked ChatGPT to. For example, a user pastes your URL and asks for a summary. OpenAI says that because these actions are initiated by a user, robots.txt rules may not apply 1.
Practice differs by company. Perplexity says Perplexity-User generally ignores robots.txt 3. Anthropic says its bots, including Claude-User, honour robots.txt 4. Google says its common crawlers always obey robots.txt when crawling automatically 2.
So if you must keep a page away from any visitor, including a bot acting for a person, robots.txt is the wrong tool. Use a login, a paywall or a server rule instead.
How much do these bots actually crawl?
Cloudflare's 2025 review found GPTBot was the second most active verified bot, at about 7.5% of verified bot traffic, behind Googlebot at over 28% 5.
It also reports OpenAI's crawl-to-refer ratio, pages crawled against visits sent back, reached as much as 3,700:1 in March 2025 5. A busy GPTBot in your logs does not mean visitors are coming.
What robots.txt rules should you write?
Write one block per bot, and keep the file small. OpenAI says changes can take about 24 hours to reach its search systems 1.
The example below allows search and refuses training. This suits a business that wants to be found but does not want its content used for training. It leaves ChatGPT-User out on purpose, because OpenAI says robots.txt rules may not apply to it 1.
- User-agent: OAI-SearchBot
- Allow: /
- User-agent: GPTBot
- Disallow: /
How do you check which bot is visiting?
Look in your server logs or CDN dashboard for the bot names, then confirm the visit comes from OpenAI's published IP ranges. Anyone can fake a user-agent name, but IP ranges are harder to fake. OpenAI publishes a JSON file of ranges for each bot 1.
Absence in your logs does not prove a block; check robots.txt and firewall rules first. Our guide to monitoring AI bot activity in server logs shows the steps, and the free AI Crawler Checker tests whether your robots.txt blocks these bots.
How should you decide what to allow?
Decide in two steps. First, ask whether you want to appear in ChatGPT answers. If yes, allow OAI-SearchBot. Second, ask whether you are happy for your content to be used for training. If no, disallow GPTBot.
The quiz below turns that into three common policies. For the same decision across all AI companies, read AI training crawlers vs AI search crawlers and how to block AI training but still appear in AI search.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. OpenAI says disallowing GPTBot signals that content should not be used to train its models 1. Appearing in ChatGPT search is governed by OAI-SearchBot, so a site can block GPTBot and still be eligible for search answers.
Is ChatGPT-User a crawler?
OpenAI says ChatGPT-User is not used for crawling the web in an automatic fashion 1. It visits pages when a user action in ChatGPT or a Custom GPT calls for it. Because a person triggered the visit, OpenAI says robots.txt rules may not apply.
How long does a robots.txt change take to work for OpenAI?
OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust 1. Wait a day before testing, and do not assume an instant effect.
Do other AI companies have the same three bots?
Can robots.txt stop ChatGPT from reading my page if a user pastes the URL?
Not reliably. OpenAI says robots.txt rules may not apply to user-initiated actions 1. If a page must stay private, put it behind a login or paywall rather than relying on a crawler rule.
Sources
- Overview of OpenAI Crawlers · OpenAIOpened October 20, 2026. Supports: Purpose of OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot; consequences of blocking; robots.txt may not apply to user-initiated actions; about 24 hours for changes; independent settings; published IP ranges.
- Google's common crawlers · Google Search CentralOpened October 20, 2026. Supports: Google-Extended is a control token for Gemini training and grounding that does not affect Search inclusion; common crawlers always obey robots.txt when crawling automatically; page last updated 2026-10-15.
- Perplexity Crawlers · PerplexityOpened October 20, 2026. Supports: PerplexityBot surfaces sites in search and is not used for foundation model training; Perplexity-User generally ignores robots.txt because the fetch is user-requested.
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic SupportOpened October 20, 2026. Supports: ClaudeBot, Claude-User and Claude-SearchBot roles; Anthropic says its bots honour robots.txt. Article dated April 7, 2026.
- Cloudflare Radar 2025 Year in Review · CloudflareOpened October 20, 2026. Supports: GPTBot about 7.5% and Googlebot over 28% of Verified Bot traffic; ChatGPT-User peak up to 16x and OAI-SearchBot about 5x the start of the year; OpenAI crawl-to-refer ratio up to 3,700:1 in March 2025.
- The rise of the AI crawler · Vercel (with MERJ)Opened October 20, 2026. Supports: Published 17 Dec 2024: none of the major AI crawlers tested, including OAI-SearchBot, ChatGPT-User and GPTBot, rendered JavaScript.