A robots.txt file is a plain-text file named robots.txt that sits at the root of a host, such as https://www.example.com/robots.txt. Use a text editor, not a word processor, and save it as UTF-8. Write User-agent groups with Disallow and Allow rules, and add a Sitemap: line with the full URL of your XML sitemap. Then upload the file to the site root and test it.
The file tells well-behaved crawlers which URL paths they may request. It is not a way to keep a page out of Google. Google's robots.txt introduction is explicit: to keep a URL out of Search, use noindex, password-protect the page, or remove it. A disallowed URL can still appear in results if other sites link to it, sometimes with only the URL and anchor text showing.
This guide follows Google's documentation for writing, uploading and testing a robots.txt file.
Where does the file have to live?
Google's guide to creating a robots.txt file sets strict location and format rules:
- The file must be named
robots.txtand must sit at the root of the host it applies to.https://www.example.com/robots.txtcontrols URLs onhttps://www.example.com/. It does not controlhttp://www.example.com/orhttps://m.example.com/, which need their own files if you use those hosts. - There can be only one robots.txt file per host.
- It must be a UTF-8 text file (ASCII is fine). Google may ignore characters outside that range, which can break rules.
- Do not create it in a word processor. Use Notepad, TextEdit, vi or a similar editor, so you do not introduce curly quotes or a proprietary format.
- If you cannot publish a file at the site root, you cannot use robots.txt for that host. Use another blocking method, such as a meta tag.
Google finds the file by requesting that root URL. You do not submit robots.txt in Search Console the way you submit a sitemap, though you can inspect the file there after it is live.
How do you write the rules?
Each group starts with one or more User-agent lines, then the paths that agent may or may not crawl. Unless you write a rule, files are allowed. Google's create-robots.txt guide lists the rules its crawlers support:
User-agent:(required). The crawler the group applies to.User-agent: *matches all crawlers except AdsBot crawlers, which must be named explicitly.Disallow:a path the crawler must not request.Allow:a path that may be crawled, used to carve out an exception inside a disallowed folder.Sitemap:the full URL of a sitemap. This line can appear anywhere in the file and can be repeated. Google does not guess the protocol or host, so write a complete URL such ashttps://www.example.com/sitemap.xml.
Comments start with #. Paths start with /. A trailing slash marks a directory. 
A simple, safe starting point for many public websites is to allow everything and point to the sitemap:
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
Only add Disallow rules for paths you truly do not want crawled, such as internal search results, filters that create infinite URL combinations, or staging folders that somehow reached the public host. Do not use robots.txt as your only protection for private files. Respectable crawlers will obey it; others may not, as Google's introduction explains.
If you also use Bing, the same file is the place to declare the sitemap, and you can confirm it after you set up Bing Webmaster Tools.





