robots.txt tester for search and AI crawlers

What robots.txt does, and what it does not

robots.txt is a plain text file at the root of your site. It tells crawlers which pages they may fetch. It is a request, not access control. Crawlers that follow the standard obey it. Other bots can ignore it.

It also works the other way. A friendly robots.txt does not mean bots get in. A firewall or CDN can still block them. Run the bot access test to see what really happens, and read our guide to firewall and CDN blocks.

How the tester reads your file

We follow the robots.txt standard, RFC 9309. In plain words:

• Each bot looks for the group that names it. Upper or lower case does not matter. If several groups name it, their rules are added together.
• A bot with a group of its own ignores the * group. A bot with no group of its own follows the * group. If there is no * group either, nothing is blocked.
• Within the group, the longest rule that matches the page wins. If an Allow rule and a Disallow rule are the same length, Allow wins.
• * stands for any run of characters. $ at the end means "the address ends here". So Disallow: /*.pdf$ blocks /files/menu.pdf but not /files/menu.pdf?v=2.
• An encoded letter or digit counts as the character itself, so /%7Eanna and /~anna are the same path.
• An empty Disallow: line blocks nothing. The file /robots.txt itself is always allowed.
• If robots.txt answers "not found" (a 4xx error), bots may fetch everything. If the server fails (a 5xx error) or does not answer, bots should fetch nothing. Google says it does not treat 429 "too many requests" as a missing file. We count 429 as a server error.

Which bots matter for AI search, and which for training

Search bots let an AI tool find your pages and use them as sources. Some other bots fetch one page because a person asked an AI tool about it. Blocking either kind can keep you out of answers. Training bots collect pages to train AI models. Blocking them is your choice, and it does not remove you from search answers. The results table groups the bots this way. Our list of AI crawlers explains each one, and the AI crawler checker tells you which blocks cost you visibility.

A worked example: our own robots.txt

This is the robots.txt that aipickcheck.com serves:

# AI Pick Check. Everything public is open to every crawler, AI search and AI user bots included.
# Private reports, the unlock steps, the API and the admin area are closed.
User-agent: *
Allow: /
Disallow: /r/
Disallow: /unlock/
Disallow: /check/
Disallow: /api/
Disallow: /admin
Disallow: /verify/
Disallow: /login
Disallow: /account

Sitemap: https://aipickcheck.com/sitemap.xml

It has one group, for every bot. It allows everything, then closes a few private areas. Here is how four paths come out, tested by the same code as the tool above:

Look at /check. The rule Disallow: /check/ ends with a slash. So it blocks the pages under /check/, but not the free check page at /check. For that page, only Allow: / matches.

Disallow: /admin has no slash at the end. So it blocks every path that starts with /admin, such as /administrator. Add the slash when you mean one folder only.

Sources

1. IETF RFC 9309: Robots Exclusion Protocol Accessed 31 October 2026
2. Google: How Google interprets the robots.txt specification Accessed 31 October 2026