Free SEO tool
Robots.txt Generator and Validator
Build a robots.txt file in a few clicks, check an existing one for mistakes, and test whether a bot can crawl any URL. Free, in your browser.
robots.txt
User-agent: * Disallow: /wp-admin/ Sitemap: https://www.example.com/sitemap.xml
Save it as robots.txt at the root of your domain (for example https://www.example.com/robots.txt).
Getting robots.txt right
One wrong line in robots.txt can hide a whole site from search, and it's a common slip after a redesign when a staging file goes live. Test your key pages here, then confirm a live page with the Google Index Checker and see AI bots in the AI Crawler Access Checker.
Fixing crawl or indexing problems? Find a technical SEO agency
Browse SEO agencies by location and service, or tell us what you need and we'll introduce you to up to three that suit your goals and budget. Free, no obligation.
More free SEO tools
- Schema Markup Generator and ValidatorBuild JSON-LD structured data, or check yours
- Meta Tag Generator and SERP PreviewPreview your search result and copy the meta tags
- Keyword Density CheckerCount the words and phrases a page repeats most
- Page Authority CheckerScore any page and see how to raise it
All 15 free tools · Markup and content
Frequently asked questions
What does robots.txt do?
It tells crawlers which parts of a site they may crawl. It sits at the root of the domain (for example https://www.example.com/robots.txt) and applies to that host only.
Does blocking a page in robots.txt remove it from Google?
No. robots.txt controls crawling, not indexing. A blocked page can still be indexed, without its content, if other pages link to it. To keep a page out of search, allow crawling and add a noindex tag.
Does Google support noindex or crawl-delay in robots.txt?
No. Google stopped supporting noindex in robots.txt in 2019 and ignores crawl-delay. Bing and some other crawlers do use crawl-delay.
How do rules apply when several match?
A crawler uses the group that names it; otherwise it uses the User-agent: * group. Within a group, the longest matching rule wins, and on a tie Allow wins. The tester follows these rules.
Should I block AI crawlers?
Blocking AI training crawlers (like GPTBot or Google-Extended) is a choice about how your content is used and doesn't affect Google Search. Blocking AI search crawlers can keep you out of answers in ChatGPT, Claude or Perplexity. Check your current settings with the AI Crawler Access Checker.