Free SEO tool
Robots.txt checker and tester
Check any website’s robots.txt in seconds. See which crawlers are blocked, from Googlebot to GPTBot, find mistakes line by line, and test any page against any crawler. Free, no sign-up.
Loading the robots.txt checker…
What the checker looks at
It reads the file exactly the way Google documents it, then explains the result in plain terms.
Every rule, read Google’s way
Each crawler follows only its most specific group. Within it, the longest matching rule wins and Allow wins a tie. Wildcards (*) and end anchors ($) are applied the same way Google applies them.
Search engines and AI crawlers
Googlebot, Bingbot and Applebot, plus GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, each shown as allowed, partly blocked or blocked, with the line that decides it.
Mistakes that cost traffic
Misspelled fields, rules before any User-agent line, full URLs instead of paths, a Noindex line Google ignores, a robots.txt that is really a web page, and a missing sitemap.
How robots.txt works
Robots.txt is a plain text file at the top of a website, for example https://example.com/robots.txt. Before a well-behaved crawler fetches a page, it reads this file to see which parts of the site it may visit.
The file is made of groups. Each group starts with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow rules. User-agent: * covers every crawler that has no group of its own. A Sitemap line can go anywhere and tells every crawler where your sitemap is.
What robots.txt does not do
- It does not remove a page from Google. A blocked page can still be indexed if other sites link to it. Use a noindex meta tag, and leave the page crawlable so Google can see it.
- It does not protect private content. The file is public, and badly behaved bots ignore it. Use a login for anything private.
- It is not read from subfolders. Each host (example.com, www.example.com, shop.example.com) needs its own file at its root.
If the file is missing or broken
A missing robots.txt (HTTP 404) means no restrictions: crawlers may fetch everything. A server error (HTTP 5xx) is more serious. Google pauses crawling the site and relies on its last good copy of the file. The checker shows which of these applies.
Robots.txt checker FAQ
How do I check my robots.txt file?
Enter your domain above and select Check robots.txt. The checker reads the live file at yoursite.com/robots.txt, flags mistakes line by line, and shows which search engines and AI crawlers are allowed or blocked.
How do I test a URL against robots.txt?
Open “Test a specific page,” enter the page path or full URL, and choose a crawler such as Googlebot. The result shows whether the page is allowed or blocked and the exact rule that decides it, using the same matching rules Google documents.
Why is my page blocked by robots.txt?
A Disallow rule matches its path. Test the page to see the exact line. Common causes are a leftover Disallow: / from a staging site and a theme or plugin rule that is broader than intended. Insights re-checks your robots.txt every week and tells you when a change blocks pages.
Does robots.txt stop a page from being indexed?
No. Robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other sites link to it. To keep a page out of search results, use a noindex tag and leave the page crawlable so Google can see the tag.
Should I block AI crawlers in robots.txt?
That is a business decision. Blocking training crawlers such as GPTBot or ClaudeBot keeps your content out of future AI training. Blocking AI search crawlers such as OAI-SearchBot or PerplexityBot also keeps you out of their answers.
The AI crawler checker shows where you stand, and the robots.txt generator builds the mix you choose.
Keep watching your robots.txt
Free tools check one thing, once. WindSpeedSEO Insights watches your whole site every week, including robots.txt and AI crawlers, and sends you a short list of what to fix next.