Free SEO tool

Robots.txt generator

Build a correct robots.txt in a minute. Start from a preset for WordPress, AI training or a staging site, add your sitemap and paths, then copy or download the file. It is checked as you type.

Loading the robots.txt generator…

How to use the file

Three steps from here to a live robots.txt.

1. Choose a starting point

Pick the preset closest to your site, add your sitemap address, and list any extra folders to keep crawlers out of, such as a cart or internal search.

2. Put it on your site

Upload the file to the top folder of your site so it opens at yoursite.com/robots.txt. On WordPress, Yoast SEO or Rank Math can edit it without uploading anything.

3. Test it

Open the result in the robots.txt checker to confirm Googlebot can reach your important pages before you rely on it.

Writing a robots.txt that does what you expect

Most robots.txt problems come from a few small mistakes. Keep these in mind when you edit the file by hand:

  • Paths are case-sensitive and must start with /. Disallow: /Cart does not block /cart.
  • A crawler with its own group ignores the User-agent: * group completely, so repeat any rules it should still follow.
  • Disallow: / under User-agent: * blocks your whole site from every search engine. Use it only on a staging site.
  • Use paths, not full URLs, in Allow and Disallow lines. Only the Sitemap line takes a full URL.
  • Do not block CSS, JavaScript or image folders that your pages need. Google renders pages and needs those files to understand them.

You can also use our SERP snippet preview tool to preview how your website pages may appear in Google search results.

Blocking AI crawlers

The “Block AI training, keep AI search” preset blocks crawlers that collect training data, plus the Google-Extended and Applebot-Extended controls, while leaving Google, Bing and AI search crawlers open. The “Block all AI crawlers” preset also blocks AI search and user-request crawlers, which removes your pages from AI answers. Search engines are never blocked by either.

Robots.txt generator FAQ

How do I create a robots.txt file?

Pick a preset above, add your sitemap and any paths to block, then copy or download the file. Upload it to the top folder of your website so it opens at https://yoursite.com/robots.txt. Each subdomain needs its own file.

What is the best robots.txt for WordPress?

For most sites, the WordPress preset: block /wp-admin/, allow /wp-admin/admin-ajax.php, and add your sitemap. WordPress SEO plugins such as Yoast and Rank Math can edit the file for you.

What does Disallow: / mean in robots.txt?

It blocks every crawler that follows robots.txt from your whole site. Use it only on a staging or test site. An empty Disallow: line means the opposite: crawlers may fetch everything.

Should I add my sitemap to robots.txt?

Yes. A Sitemap line tells every search engine where your sitemap is, including ones you have not registered with. It takes a full URL, and you can list more than one.

How do I block a page or folder in robots.txt?

Add Disallow: /that-page/ or Disallow: /that-folder/ under User-agent: *. If you want the page out of Google’s results, use a noindex tag instead, because a blocked page can still be indexed from links.

Let us keep it right

Free tools check one thing, once. WindSpeedSEO Insights watches your whole site every week, including robots.txt and AI crawlers, and sends you a short list of what to fix next.