Tool 1 of 8 · Let AI assistants in
AI crawler checker
Paste your website address. We read your robots.txt and tell you, for each AI crawler, whether it is allowed in, what it is for, and whether it obeys the file at all.
- Free, no sign-up
- About 5 seconds
- We do not keep the pages we read
Open doors are only half of it
robots.txt says who may come in. It does not say whether assistants then recommend you. The free check asks real buyer questions and shows whether your business is named.
Next tool: Bot access test — Opens your page the way each key AI bot does, and as a normal browser, to catch firewall and CDN blocks.
Questions people ask
Should I block AI training bots?
That is up to you. Blocking a training bot tells that company not to use your pages to train its models. It does not stop assistants from finding you through their search bots, so it does not cost you recommendations.
What happens if I block a search bot?
That company's assistant cannot use your pages as a source. Some company pages say a blocked page can still show up as a bare link. If you want to be recommended, allow the search bots.
Why does a bot with its own group ignore my "*" rules?
Under the robots.txt standard, a crawler follows the group that names it and ignores the "*" group. If you add a group for one bot, copy over any rules you still want it to follow.
Does a clean robots.txt mean assistants can read my site?
No. A firewall or CDN can still turn bots away. The bot access test checks that.