Free SEO tool
AI crawler checker
See whether ChatGPT, Claude, Perplexity, Gemini and other AI crawlers can read your website. Check training and AI search crawlers separately, from your live robots.txt or a pasted file.
Loading the AI crawler checker…
Three kinds of AI access
AI companies use separate crawlers for different jobs, so you can allow one and block another.
Training crawlers
GPTBot, ClaudeBot, CCBot and others collect content that may be used to train AI models. Blocking them keeps your future content out of training.
AI search crawlers
OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether your pages can appear and be cited in AI answers. Blocking them removes you from those answers.
User requests
ChatGPT-User, Claude-User and Perplexity-User visit a page when a person asks about it. Some of these say robots.txt may not apply to them.
Blocking AI without hurting Google
Google Search and Google’s AI training are controlled separately. Googlebot crawls for Google Search. Google-Extended is not a crawler at all: it is a token that tells Google whether content may be used to train Gemini models and for grounding. Blocking Google-Extended does not affect your Google Search rankings. Apple works the same way with Applebot and Applebot-Extended.
A common setup blocks training crawlers and keeps search and AI search open:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
Group several crawlers above one set of rules like this, and leave your User-agent: * group as it is. The generator writes this for you.
What blocking cannot do
- It does not remove content already collected. It applies to future crawling.
- It relies on each company honoring robots.txt. The major AI companies say their crawlers do, apart from the user-request agents noted above.
- New crawlers appear regularly, so a file written today may not name next year’s bots.
AI crawler checker FAQ
How do I block AI crawlers like GPTBot and ClaudeBot?
Add a group for each crawler to your robots.txt, for example User-agent: GPTBot followed by Disallow: /. The robots.txt generator writes these rules for you, and this checker confirms they work.
Will blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. GPTBot is OpenAI’s training crawler. Whether your pages appear in ChatGPT search answers depends on OAI-SearchBot, which you can allow separately.
Does blocking AI crawlers hurt my Google rankings?
No, as long as Googlebot stays allowed. Google states that Google-Extended does not affect inclusion or ranking in Google Search. It only controls use of your content for Gemini training and grounding.
Do AI crawlers respect robots.txt?
The major AI companies say their crawlers honor robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person asked for the page.
How can I see if ChatGPT mentions my business?
Robots.txt only controls access. To see whether ChatGPT, Claude or Perplexity actually mention and cite you, AI search visibility tracking runs a fixed set of questions and records the answers over time.
Know where you stand in AI search
WindSpeedSEO Insights checks your robots.txt for search and AI crawlers every week and tracks whether AI assistants mention your business.