Whether to block AI crawlers depends on what you want from AI tools. Blocking AI training crawlers (such as GPTBot and ClaudeBot) opts your content out of future model training and has no effect on Google rankings. Blocking AI search crawlers (such as OAI-SearchBot, Claude-SearchBot and PerplexityBot) can keep you out of those tools' answers and citations. Most businesses that want to be found should allow the search crawlers and make a deliberate choice about the training ones.
This article explains the different kinds of AI bot, what each major company says its bots do, and how to set robots.txt rules for the choice you make.
Not all AI bots do the same job
AI companies now document their bots in detail, and most split them into three groups.
Training crawlers
These collect public web pages that may be used to train future AI models. Blocking them is a signal that your content shouldn't be used for training. Examples: OpenAI's GPTBot, Anthropic's ClaudeBot and Common Crawl's CCBot, whose open archive is widely used for training.
Search crawlers
These build the index an AI assistant searches when it answers questions, so they decide whether your pages can be found, quoted and linked. Examples: OpenAI's OAI-SearchBot (ChatGPT search), Anthropic's Claude-SearchBot and PerplexityBot.
User-requested fetchers
These visit a page because a person asked the assistant to look at it, for example by pasting a link. Examples: ChatGPT-User, Claude-User and Perplexity-User. Because a human started the request, OpenAI and Perplexity both say robots.txt rules may not always apply to these visits.
Control tokens
Some names in robots.txt aren't bots at all. Google-Extended and Applebot-Extended are tokens that let you say whether content Google or Apple already crawls may be used for their AI models. Crawling itself is still done by Googlebot and Applebot.

What each company says
OpenAI
OpenAI says its settings are independent. GPTBot crawls content that may be used to train its models, and disallowing it signals that your content shouldn't be used for training. OAI-SearchBot is used to show websites in ChatGPT's search features; sites that block it won't be shown in ChatGPT search answers, though they may still appear as plain navigational links. OpenAI recommends allowing OAI-SearchBot if you want to appear, and says robots.txt changes for search can take about 24 hours to take effect.
Anthropic
Anthropic runs three bots. ClaudeBot collects content that may be used for training; blocking it excludes your future content from training data. Claude-SearchBot indexes content to improve Claude's search results, and Claude-User fetches pages when users ask. Anthropic says blocking either of the last two may reduce your visibility in Claude's answers. It also supports the non-standard Crawl-delay rule, and says you need to set rules on each subdomain you want to cover.
Perplexity
Perplexity says PerplexityBot is designed to surface and link websites in its search results and isn't used to crawl content for AI foundation models. It recommends allowing it if you want to appear. Perplexity-User handles user requests and, Perplexity says, generally ignores robots.txt because a person asked for the page.
Google-Extended controls whether content Google crawls may be used to train future Gemini models and to ground answers in the Gemini apps and some Google Cloud products. Google says it doesn't affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search, so they're governed by Googlebot and the normal snippet controls, not by Google-Extended.






