In more detail

OpenAI documents three main agents that matter to most site owners:

  • GPTBot crawls content that may be used to train OpenAI's foundation models. Disallowing it signals that your content shouldn't be used for training.
  • OAI-SearchBot is used to surface websites in ChatGPT's search features. OpenAI says sites that block it won't be shown in ChatGPT search answers, though they can still appear as plain navigational links. It recommends allowing it if you want to appear.
  • ChatGPT-User fetches a page when a ChatGPT user's request needs it. It isn't an automatic crawler, and OpenAI notes that robots.txt rules may not apply to these user-initiated visits.

Each setting is independent, so a common choice is:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

That opts out of training while staying eligible for ChatGPT search. OpenAI says changes for search can take around 24 hours to take effect, and it publishes the IP ranges each bot uses so you can verify genuine visits in your server logs.

Remember that a bot with no group of its own follows your User-agent: * rules. A site that disallows everything for * and only allows Googlebot is blocking GPTBot and OAI-SearchBot too.

Why it matters when hiring an agency

Whether to block GPTBot is a business decision about how your content is used, not an SEO trick, and blocking it has no effect on Google rankings. Blocking OAI-SearchBot is different: it removes you from ChatGPT search answers. An agency should explain that trade-off clearly and check your current rules before suggesting changes. Check yours with our AI crawler access checker.