AI crawlers explained: training, search and user bots

The short answer

Most big companies run more than one bot, and each bot has one of three jobs. A search bot builds an index that answers draw on. A user-fetch bot opens one page because a person asked. A training bot gathers material for models. You can allow or block them one by one in robots.txt. The user-fetch bots are the awkward ones, because several companies say they may skip robots.txt.

Everything below is what each company says on its own page, read on 2026-10-15. Bot names and rules change, so treat this as a snapshot and check the linked pages before you act.

The table

Company by company

OpenAI

OpenAI describes OAI-SearchBot as the bot that surfaces sites in its search features, and GPTBot as the bot that gathers content that may be used for training. Each setting is independent, so you can allow search and refuse training. ChatGPT-User handles visits a person starts, and OpenAI says it does not decide whether you appear in search. OpenAI also lists OAI-AdsBot, which only visits pages submitted as ads. Its page does not say whether robots.txt applies to that one.

Anthropic

Anthropic names three bots: ClaudeBot (training), Claude-SearchBot (search relevance) and Claude-User (fetching a page when a user asks). It says they follow industry-standard robots.txt directives and that it supports the non-standard Crawl-delay line. It advises against blocking by IP address, because that stops the bots reading your robots.txt.

Perplexity

Perplexity says PerplexityBot is for showing and linking sites in search results and is not used to train foundation models. Perplexity-User fetches a page when a person asks and generally ignores robots.txt.

Google

Googlebot is Google's main crawler, and Google says its common crawlers always obey robots.txt when they crawl automatically. Google-Extended is different. Google calls it a standalone product token and says it has no separate user agent string. You use it in robots.txt to say whether content Google has crawled may be used for future Gemini model training and for grounding in Gemini Apps and Vertex AI. Google says it does not affect your inclusion in Google Search. Google's overview also describes user-triggered fetchers, but the page does not say whether they follow robots.txt.

Apple

Applebot powers search in Spotlight, Siri and Safari, and may also help train Apple's models. If your robots.txt has no Applebot rules but has Googlebot rules, Apple says Applebot follows the Googlebot rules. Applebot-Extended does not crawl at all. Disallowing it only limits how data Applebot already collected may be used for training.

Meta

Meta lists meta-webindexer (Meta AI search), meta-externalagent (training and indexing) and meta-externalfetcher (fetching a link at a user's request, which "may bypass robots.txt"). It also says facebookexternalhit, the link-preview fetcher, might bypass robots.txt for security or integrity checks. Meta says crawlers may cache your robots.txt for up to 24 hours.

Microsoft and Bing

We could not open Bing's crawler documentation today, so we make no claim about Bingbot's rules here. Cloudflare lists BingBot among crawlers that serve both search and training purposes, which matters if you use its bot settings (see our firewall guide).

Common Crawl

Common Crawl is a non-profit that keeps an open archive of web pages. Its bot is CCBot. It says you can block it with a robots.txt rule, and warns that other bots pretend to be CCBot, so check the IP before you trust a request.

A robots.txt that blocks training and keeps search

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Leave out the search bots, so they keep their access. Read our first guide before you add an Allow group for any of them.

How to check it worked

Use the free crawler checker at /tools/ai-crawler-checker. It reads your robots.txt and shows, bot by bot, what is allowed. Changes can take up to a day: OpenAI, Perplexity and Meta each mention about 24 hours.

What this won't do

• A robots.txt rule is a request. It does not stop a bot that ignores it.
• Blocking a training bot does not remove content that was already collected.
• Blocking Google-Extended does not keep you out of Google Search, and allowing it does not put you into answers.
• User-fetch bots may still reach a page you blocked.
• Companies change bot names and rules. This table will age.