Check that AI search bots can reach your website

The short answer

Open yourdomain.com/robots.txt and look for Disallow: / under a crawler name or under User-agent: *. If you find one, a crawler may be shut out of your whole site. Then ask whoever runs your hosting whether a firewall also turns bots away. Both checks take about ten minutes.

If an AI assistant cannot read your site, it may have to rely on what other sites say about you, and that can be out of date. Nobody can promise that being readable makes an assistant recommend you. Being unreadable removes one of your chances.

Why the bot's job matters

The companies behind AI assistants publish the names of their crawlers. Most run several, and each has its own job:

• a search crawler builds an index of pages so they can be used in answers;
• a user-fetch crawler opens one page because a person asked about it;
• a training crawler gathers material to train models.

You control them separately. Blocking a training crawler is a choice about training. Blocking a search crawler can mean your site is left out of answers. Our crawler guide lists them company by company.

Step 1: read your robots.txt

Look for lines that start with User-agent: followed by a name, and the rule under them.

• Disallow: / under a name means that crawler may not read anything.
• User-agent: * with Disallow: / shuts out every crawler that has no group of its own.
• A crawler with a group of its own follows that group only. Google documents this for its crawlers: only the most specific matching group applies, and it is not merged with the * group.

Also check the address. A robots.txt file only covers the host it sits on, so a subdomain needs its own file.

Step 2: decide what you want, then write it down

What the companies say, as of the date in the sources:

• OpenAI says OAI-SearchBot is for search and that sites which opt out will not appear in its search answers. GPTBot is for training, and blocking it does not affect search. ChatGPT-User acts when a person asks, and OpenAI says robots.txt rules may not apply to it.
• Perplexity says PerplexityBot surfaces and links sites in its search results, and recommends allowing it in robots.txt. It says Perplexity-User generally ignores robots.txt because a person asked for the page.
• Anthropic names ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests. It says its bots honour robots.txt.

A common choice for "be found, but do not train on me" is to block the training crawlers and leave the others alone:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Be careful with a bare Allow: / group for one crawler. That crawler then ignores the * group, so any private folders you hid there are open to it. Repeat your private rules inside the new group.

Step 3: check the firewall

A friendly robots.txt does not help if a firewall or CDN answers the bot with an error. Ask your host or developer to look in the logs for the crawler names above. Many 403 or 429 answers mean the bot is being refused. Our firewall guide shows what to ask for.

Anthropic advises against blocking its crawlers by IP address, because then they cannot even read your robots.txt to see what you want.

Step 4: check the page without JavaScript

Vercel looked at the traffic of several AI crawlers and reported in December 2024 that the bots from OpenAI, Anthropic, Perplexity and Meta did not run JavaScript, while Google's and Apple's did. If your price or main offer appears only after a script runs, some crawlers may never see it. That study is old, so test your own pages. See the JavaScript guide.

A quick example

A fictional bakery, Maple Row Bakery at example.com, finds this in its robots.txt:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

The writer meant "block the junk bots, keep Google". The result is that every crawler without its own group is shut out, including search bots from the AI companies. The fix is to name the bots you want to allow, or to remove the blanket rule and block only the specific ones you do not want. After the change the bakery reads the file again and runs the crawler checker.

Files like this can come from a staging site that was copied to the live one. If your developer launched the site recently, ask whether a "block everything" rule was left behind.

How to check it worked

1. Open yourdomain.com/robots.txt after you save it, and read it as a visitor would.
2. Run our free crawler checker at /tools/ai-crawler-checker. It reads your robots.txt and tells you which crawlers it allows.
3. Run the bot access test at /tools/bot-access-test to see whether your server answers bots normally.
4. Wait. Perplexity says a robots.txt change can take up to a day to be picked up, and OpenAI says the same for its search results. Note the date you changed it.

What this won't do

These limits are worth knowing before you spend time on this.

• It will not make any assistant mention you. Access is a precondition, not a ranking.
• It will not stop a bot that ignores robots.txt. The file is a request, not a lock.
• It cannot control user-fetch bots that the company says may not follow robots.txt.
• It tells you nothing about how good your content is.
• It is not a one-time job. Hosts, plugins and firewalls change settings, so look again every few months.