AI Crawler Checker
Enter your website. See which AI bots your robots.txt allows or blocks, whether your homepage can be read without JavaScript, and what to fix first. Free, with no sign-up.
How it works
- We fetch your homepage,
robots.txt,llms.txtand sitemap, one request each. We do not save what we fetch. - We read your
robots.txtthe way the standard describes. Then we report what it says about each named AI and search crawler, such as OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot. - We look at the raw HTML of your homepage. That is the text a crawler gets before any JavaScript runs. We check for readable text, a title, a description, structured data and a noindex tag.
- You get a plain verdict, a table of bots, a readiness list and a fix list with copy-ready snippets. A builder at the end writes a
robots.txtblock from your choices. - Search bots and training bots are different. You can allow one and block the other. See training bots vs search bots.
What it cannot tell you
robots.txtshows what you allow, not what a bot does. We cannot tell you whether any AI bot has visited your site. Your server logs can: see how to monitor AI bot activity in server logs.- A CDN, firewall or bot-protection service can still block AI bots when your
robots.txtallows them. We cannot see those rules from outside. - We fetch your homepage only, and we read its HTML only, not its HTTP headers. "Readable" is judged from the raw HTML text. Google can run JavaScript, so a page that looks empty here may still work for Google.
- We cannot predict citations. Being readable is a requirement, not a reason for an AI to cite you. Content quality, trust and relevance decide that.
Questions
What is an AI crawler?
An AI crawler is a program that fetches web pages for an AI company. Some collect pages to train models. Some build a search index so the assistant can cite pages. Some fetch a page only when a person asks the assistant to open it. The first kind is a training bot, the second a search bot and the third a user bot.
Which bots should I allow if I want to be cited in AI answers?
The search and user bots: OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User for Perplexity, Claude-SearchBot and Claude-User for Claude, plus Googlebot and Bingbot. Allowing them does not mean you will be cited. It only removes one reason you might not be. See how to allow AI crawlers in robots.txt.
Should I block AI training bots?
That is your choice. Training bots such as GPTBot and ClaudeBot collect pages that may be used to train models. Blocking them does not remove you from AI search answers, because those use separate bots. Training access does not decide whether you are cited in AI search answers. Decide based on how you want your content used. See how to block AI training but allow AI search.
My robots.txt allows everything. Why might AI still not use my site?
robots.txt is only one gate. A firewall or CDN can refuse the bots. Your content may appear only after JavaScript runs. The page may carry a noindex tag. Or the page may not yet be useful or trusted enough to cite. This checker looks at the first three from the outside, but it cannot see inside your firewall.
Does this tool save my website or results?
No. We fetch the pages once, show you the result and do not store it. To limit abuse, we keep a short-lived count of requests per network address in memory. In your server logs, the check appears as ChalkLabsAISEOBot/1.0.
Built by
Rahil Jain
Rahil built this tool for the same method Chalk Labs uses on client work. It is free, and it is not a sales funnel: nothing here requires an email.