Robots.txt generator and testing guide

Build a conservative starting file, then check every rule before publishing. Robots.txt controls crawling; it is not a reliable way to remove a page from search.

Your site URL Platform Add example AI crawler blocks (verify names first)

Your robots.txt

How to use a robots.txt file safely

A robots.txt file gives crawl instructions for one hostname and protocol. The file for www.example.com does not automatically control shop.example.com, and an HTTP file is not a substitute for the HTTPS version. Keep the file short enough that another person can explain every line.

Use disallow rules to manage crawler access to repetitive or operational URL spaces, not to hide confidential information. The file is public, rules are crawler directives rather than access control, and a blocked page may still appear as a URL-only result when other pages link to it.

Robots.txt versus noindex

Use noindex when a publicly accessible HTML page should not appear in supported search results. The crawler must be allowed to fetch the page to see that directive. Blocking the URL in robots.txt at the same time can prevent the crawler from reading the noindex instruction.

For private customer data, staging systems or admin areas, use authentication and appropriate server security. Neither robots.txt nor a meta tag makes private content private.

Platform cautions

WordPress normally needs very few custom rules; blocking theme, plugin or media assets can interfere with rendering. Shopify generates a platform robots file and advanced changes use its theme-level robots.txt mechanism. Hosted builders may manage the file for you. Read the platform’s current documentation before replacing an existing file.

Validate the generated file

Open the intended sitemap and confirm it returns XML successfully.
Test the homepage and priority service pages against the rules.
Inspect staging, search, filter, cart and admin paths separately.
Deploy the smallest change, then monitor crawling and indexing evidence.

Google’s robots.txt introduction explains the boundary between crawl control and index control. Use the generated text as a reviewed starting point, not a universal configuration.

Frequently asked questions

Where does robots.txt go? +

Publish it at the root of the hostname, for example https://www.example.com/robots.txt. A file in a subfolder does not control the whole site.

Can robots.txt remove a page from Google? +

No. Robots.txt controls crawler access, not reliable removal from search. A disallowed URL can still be indexed from links elsewhere. Allow crawling and use a noindex meta tag or X-Robots-Tag when removal is the goal.

Can I block AI crawlers? +

You can add user-agent rules for named crawlers that publish and honour robots controls. Bot names and policies can change, so verify each operator’s current documentation before publishing a block list.

Should every website have disallow rules? +

No. A small public website may need only a User-agent line, an Allow rule and a valid sitemap reference. Add disallow rules only for crawl paths you understand; robots.txt is not a generic SEO optimisation checklist.