In more detail
AI companies now publish documentation for their bots, and most split them into three jobs:
- Training crawlers collect content that may be used to train future models. Examples: OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot.
- Search crawlers build the index an AI product searches when it answers questions. Examples: OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot, PerplexityBot.
- User-triggered fetchers visit a page when a person asks the assistant to look at it. Examples: ChatGPT-User, Claude-User, Perplexity-User. Because a person asked, some of these say robots.txt rules may not always apply.
Some names are control tokens rather than separate bots. Google-Extended and Applebot-Extended don't crawl on their own: they let you say whether content Google or Apple already crawls may be used for their AI models.
You control most of these with robots.txt, using a group for each user-agent token. The rules are read like Googlebot's: a bot follows its own group if there is one, and otherwise the User-agent: * group. Blocking a training crawler and allowing a search crawler are separate choices, so you can opt out of training without disappearing from AI search answers.
Firewalls and CDN "bot fight" settings can also block AI crawlers without anyone editing robots.txt, so check both.
Why it matters when hiring an agency
Many sites block AI search crawlers by accident, through an old catch-all rule or a security setting, and then wonder why they never appear in ChatGPT or Perplexity answers. A competent agency should check this early, explain the difference between training and search bots, and help you make a deliberate choice. Our free AI crawler access checker shows which AI bots your robots.txt allows.