Free AI SEO tool
AI Crawler Access Checker
See which AI bots your robots.txt lets in, and which it keeps out. Free, no sign-up.
Can AI search tools reach your site?
ChatGPT, Claude, Perplexity and other AI assistants use their own crawlers to find pages they can quote and link to. Each company also runs separate bots that collect content for training its models. Your robots.txt file decides which of them may crawl your site, and one old catch-all rule can block them all without anyone noticing.
This tool reads your robots.txt the way the crawlers do and shows the result for each AI crawler, grouped by what it's for, so you can make a deliberate choice: stay visible in AI search, opt out of training, or both.
Four kinds of AI bot
AI search
Builds the index an AI assistant searches before it answers. Blocking it can keep you out of that tool's answers and citations.
Model training
Collects pages that may be used to train future AI models. Blocking it opts out of training and has no effect on Google rankings.
AI use control token
Not a separate crawler: a robots.txt token that says whether content the company already crawls may be used for its AI models.
User-requested fetch
Visits a page when a person asks the assistant to. The operator says robots.txt may not always apply to these visits.
More free AI SEO tools: llms.txt Generator and Validator and AI-Readiness Checker. Guides, glossary and agencies in one place: the AI SEO hub.
Want help with AI search visibility? Find an AI SEO agency
Browse agencies that list AI SEO as a service, or tell us what you need and we'll introduce you to up to three that suit your goals and budget. Free, no obligation.
Frequently asked questions
Should I block AI crawlers?
It depends what you want. AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether your pages can be found and cited in those tools' answers, so most businesses that want customers to find them should allow them. AI training crawlers (GPTBot, ClaudeBot, CCBot) collect content that may be used to train future models; blocking them is a choice about how your content is used and doesn't affect Google rankings.
Does blocking Google-Extended keep me out of AI Overviews?
No. Google says Google-Extended controls whether content it crawls may be used to train Gemini models and to ground Gemini apps. It doesn't affect inclusion or ranking in Google Search, and AI Overviews and AI Mode are part of Search. To limit how a page appears there, use nosnippet, max-snippet or noindex.
Why does a bot show as blocked when I never mentioned it?
A crawler with no group of its own follows your User-agent: * group. If that group disallows the whole site, every AI bot you haven't named is blocked too. Give each bot you want to allow its own group.
What does 'partly blocked' mean?
The bot may crawl the URL you entered, but robots.txt blocks it from at least one common section of the site, such as the homepage, /blog/ or /services/. Check the rules to make sure that's deliberate.
Robots.txt says allowed, but the bots still can't get in. Why?
robots.txt is only one layer. A firewall, CDN bot protection (for example Cloudflare's AI bot blocking) or IP rules can block crawlers that robots.txt allows. Check your hosting and CDN security settings, and your server logs for 403 responses to these user agents.
Do AI bots actually obey robots.txt?
The major AI companies say their automatic crawlers do, and publish the user-agent names to use. Some fetchers that act on a user's direct request, such as ChatGPT-User and Perplexity-User, say robots.txt may not always apply. Less reputable scrapers may ignore it entirely.
FindSEO isn't affiliated with or endorsed by any of the companies named. Bot names and purposes are taken from each company's own documentation, linked from the results table. This tool only fetches the site's homepage and robots.txt.