Robots.txt generator

Choose what search engines and other crawlers may visit, add your sitemap and copy the file. The tester below says whether a crawler may fetch a given address under the rules you wrote.

Start from
By default
robots.txt 
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search

Sitemap: https://example.com/sitemap.xml

Save it as robots.txt in the root folder of your site, so that it opens at https://your-domain/robots.txt. A robots.txt anywhere else is ignored.

Hosting your own site? Full control over files like this one comes with any VPS. From €3.02 a month today, among 2,226 servers.

Compare small servers Redirect generator

Test an address

Googlebot Blocked Decided by line 3: Disallow: /cart/

Group used: User-agent: *

Works in your browser: nothing you type is sent anywhere.

How it is calculated

A robots.txt is made of groups. Each group starts with one or more User-agent lines naming the crawlers it is for, followed by Allow and Disallow rules. A crawler follows only one group: the one that names it most precisely, or the User-agent: * group when none names it.

# the group for every crawler
User-agent: *
# no address that starts with /admin/
Disallow: /admin/
# except this one
Allow: /admin/help

Sitemap: https://example.com/sitemap.xml

Within the group, the rule with the longest path that matches the address wins, and Allow wins a tie. The tester applies exactly these rules, as written in RFC 9309 and followed by Google and Bing, and shows the line that decided.

Example

With Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php, Googlebot may not fetch /wp-admin/options.php, because only the Disallow rule matches. It may fetch /wp-admin/admin-ajax.php: both rules match, and the Allow rule is longer.

Questions

Does robots.txt keep a page out of Google?

Not reliably. Disallow stops crawling, but Google can still list a blocked address it finds through links, without a description. To keep a page out of search results, let it be crawled and add a noindex robots meta tag or X-Robots-Tag header, or protect it with a password.

Where do I put the robots.txt file?

In the root of the host, so that it opens at https://example.com/robots.txt. Each host needs its own: www.example.com, shop.example.com and http and https versions are read separately. Crawlers ignore a robots.txt in a subfolder.

How do I block AI crawlers like GPTBot and ClaudeBot?

Give them a group of their own with Disallow: /, as the generator does. Google-Extended and Applebot-Extended are tokens rather than crawlers: blocking them stops Gemini and Apple from training on your pages without leaving Google or Apple search. Robots.txt is a request, so a crawler that ignores it has to be blocked on the server.

Is robots.txt a security measure?

No. Anyone can read the file, and it lists the paths you would rather hide. Polite crawlers follow it, others do not. Keep private areas behind a login or a password, for example with HTTP basic authentication.