Blocking AI bots: what you gain and what you lose

"Block the AI bots" sounds like one switch. It is at least three, and they do different things. Flip the wrong one and you can disappear from the answers you wanted to be in. Here is how to think it through.

The short answer

Blocking training bots is a choice about how your content is used to build models. Blocking search bots can remove you from AI answers. Blocking user-request fetchers is the least reliable, because some companies say their rules may not apply. Decide each one separately.

The three kinds of bot

Cloudflare, which classifies bots that visit its network, uses similar groups. It describes "AI Crawler" bots as gathering content to train models, "AI Search" bots as powering AI search, and "AI Assistant" bots as running only when a user triggers them.

Training bots. They collect content that may be used to train models. Examples the companies name: GPTBot (OpenAI), ClaudeBot (Anthropic) and the Google-Extended control token (Google).

Search bots. They build the index an assistant searches. Examples: OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot (Perplexity). Google's own crawler, Googlebot, feeds Google Search, and Google says its AI features draw on that index.

User fetchers. They fetch a page because a person asked an assistant something. Examples: ChatGPT-User, Claude-User and Perplexity-User.

What each company says

• OpenAI. GPTBot is for content that may be used to train its models, and blocking it tells OpenAI not to use your content for that. OAI-SearchBot is for ChatGPT search, and OpenAI says blocked sites will not appear in its search answers, though they can still show as links. ChatGPT-User acts when a user asks, and OpenAI says robots.txt rules may not apply to it.
• Anthropic. ClaudeBot collects content that could be used for training. Blocking Claude-SearchBot means your content is not indexed for its search, which may reduce your visibility. Blocking Claude-User means pages cannot be fetched for a user's question, which may reduce visibility in user-directed search. Anthropic also warns that blocking IP addresses may not work, because it stops the bot from reading your robots.txt.
• Perplexity. PerplexityBot surfaces sites in its search and is not used to train foundation models. Perplexity recommends allowing it. It says Perplexity-User generally ignores robots.txt because a person asked for the page.
• Google. Common crawlers always obey robots.txt for automatic crawls. Google-Extended is a control token for whether content may be used for future Gemini model training and for grounding in the Gemini apps and Vertex AI. Google says it does not affect a site's inclusion in Google Search. For its generative AI features, Google says a page must be indexed and eligible to be shown in Google Search with a snippet.

Decision table

"Probably" is deliberate. The companies describe the bots. They do not publish how much any one rule changes your results.

Which way to lean

You sell to the public and want to be found. Allow the search bots. Decide the training question as a separate choice, since the companies above document their training and search bots separately.

Your content is the product (paid articles, courses, data). Blocking training bots is a reasonable choice. Remember that robots.txt is a request, not a lock: the standard itself says its rules are "not a form of access authorization". A login or paywall is the lock.

You are not sure. Start by allowing the search bots and blocking nothing else. Revisit in a few months.

A starting robots.txt

This example blocks the training bots only. It uses a fictional site.

# https://www.example.com/robots.txt
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Search bots and everyone else: no rule means allowed
User-agent: *
Disallow:

Test any change before you rely on it. The robots.txt standard says a crawler finds the group that matches its own name, and falls back to the * group only when none does.

How to check it worked

1. Open example.com/robots.txt in a browser and read it back.
2. Allow up to 24 hours. OpenAI and Perplexity both say robots.txt changes can take that long.
3. Look at your firewall or CDN settings. Bot protection can turn crawlers away even when robots.txt allows them; the guide When your firewall or CDN turns AI bots away shows how to test. Perplexity advises matching both the user-agent and its published IP ranges if you allowlist.
4. Follow the steps in Check that AI search bots can reach your website.

What this won't do

Anthropic describes blocking as excluding your site's future materials, so do not expect it to undo what was already collected. Robots.txt is a request that well-behaved bots follow, and some fetchers say they do not. Allowing the search bots does not mean an assistant will name you. And bot names and rules change, so recheck the companies' pages before you act.