When your firewall or CDN turns AI bots away

The short answer

Your robots.txt can say "come in" while your firewall says "go away". Bot protection works on its own rules, before robots.txt is even read. If it blocks or challenges an AI crawler, that crawler gets an error or a "prove you are human" page instead of your content. You can find this out with a free test and fix it in a few settings. Do not switch protection off. Allow the specific bots you want.

This guide uses Cloudflare as the worked example because it is common and its documentation is public. Other firewalls and hosts have similar switches under different names.

What can go wrong

• A setting that blocks AI bots as a group. Cloudflare's AI Crawl Control lets you set allow or block rules for individual crawlers, and says the blocks are enforced by a WAF rule it creates on your site. A block you set once and forgot will still apply.
• New defaults. Cloudflare's July 2026 post sorts AI bots into three groups: Search, Agent and Training. It says that from 15 September 2026, new domains joining Cloudflare have Agent and Training crawlers blocked by default on pages that show ads, while Search stays allowed. It says site owners can change this in their Security settings. The post does not say clearly how existing sites are handled, so look at your own dashboard.
• Mixed-purpose bots. Cloudflare says bots that do both search and training, such as Googlebot, Applebot and BingBot, are judged by the most restrictive applicable rule, so a customer who blocks Training also blocks those bots. Blocking "training" the wrong way could therefore cost you ordinary search crawling.
• Challenges. Bot Fight Mode issues challenges meant to make automated requests costly. Cloudflare says its Skip, Bypass and Allow rules have no effect on Bot Fight Mode, and that the version that supports exceptions is Super Bot Fight Mode. A challenge page is useless to a crawler.
• "Verified" does not always mean "allowed". Cloudflare's docs say verified bots have historically been left out of its default bot settings. Its 2026 post says the Verified label now only makes a bot allowable within its own category. The docs also say a service can lose its verified status if it breaks the requirements, for example an AI crawler that ignores a robots.txt crawl-delay.

Step 1: test what a bot sees

Run our bot access test at /tools/bot-access-test. It opens your page as a normal browser, then once for each main AI bot name, and compares what comes back. Look for:

• a 403 (refused) or 429 (too many requests) for a bot name;
• a short page with a title like "Just a moment" or "Attention required", which is a challenge;
• a big difference in page size between the browser and a bot.

Be aware of the limit. The test sends each bot's name from our server, not from the bot's real network address. A firewall that checks the real address may treat us differently from the real bot. So a pass is a good sign, not a proof.

Step 2: look at your own logs

Ask your host or developer for the last week of requests that mention OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot or Googlebot. Check the status codes. A run of 200 answers is healthy. Many 403 or 429 answers means something is turning them away.

Step 3: fix it safely

Aim for rules that let in the bots you want and nothing else.

1. Decide which bots you want. Search bots are the usual yes. Your choice on training bots is separate (see the crawler guide).
2. Use per-bot allow rules. In Cloudflare, go to AI Crawl Control, open the Security tab, and set Allow or Block for each crawler in the table.
3. Match name and address together. Perplexity's own advice for firewalls is to require both the user-agent name and an address from its published list, and to refresh the list because it changes. OpenAI and Anthropic also publish address lists. A name alone can be faked by anyone.
4. Put allow rules above block rules. In its AWS firewall example, Perplexity says to give allow rules a higher priority than your blocking rules. The same idea applies on other firewalls.
5. Keep protection on for everything else. Do not turn off bot protection for the whole site just to let a few crawlers in.
6. Do not block by address only. Anthropic warns that blocking its IP addresses stops its bots reading your robots.txt.

A short note to send your developer:

Hi. Please check whether our firewall or bot protection blocks or
challenges these crawlers on example.com: OAI-SearchBot, ChatGPT-User,
PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Googlebot.
If it does, please add allow rules that match both the user-agent name
and the address ranges the companies publish, placed above any block
rules. Please keep bot protection on for all other traffic. Thanks!

How to check it worked

1. Run the bot access test again and compare.
2. Run the crawler checker to confirm robots.txt still allows what you chose.
3. Look at your logs after a few days. You should see 200 answers for the bots you allowed.

Allow time before you judge a robots.txt change: Perplexity says it can take up to 24 hours. A firewall change may have its own delay, so ask your host.

What this won't do

• It will not make an assistant name you. It only removes one reason a bot could fail to read you.
• The test cannot copy the real bot's address, so it can miss address-based blocks.
• Allowing a name is not the same as being safe. Match the address too.
• Companies change names, defaults and dashboards. Check the date on this page.