llms.txt is a plain Markdown file at the root of a website that gives AI agents a short description of the site and a hand-picked list of links to its most useful pages. It takes an hour or two to create, it does no harm, and it can help AI tools that look for it. But it isn't a ranking factor, Google says it doesn't need it, and it won't fix a site that AI crawlers can't reach.
Here's what the file is, who actually reads it, how to write one, and how much effort it deserves.
What is llms.txt?
The idea was proposed by Jeremy Howard in September 2024 and revised in 2026. Web pages are built for people, wrapped in menus, scripts, cookie banners and adverts. An AI agent trying to help someone has to dig through all of that, and it has a limited amount of space to hold what it reads. llms.txt gives it a shortcut: a concise overview and a map of where the important information lives.

The proposal compares it to robots.txt and XML sitemaps, but it does a different job:
- robots.txt tells crawlers what they're allowed to access.
- An XML sitemap lists every page you want search engines to index.
- llms.txt is a short, curated guide for an agent that needs to understand your site quickly.
It doesn't block or allow anything, and it doesn't replace either of the other two.

Who actually reads it?
This is where expectations need managing.
According to the proposal's author, thousands of sites now publish an llms.txt, documentation platforms generate one automatically, and the major AI companies publish llms.txt files for their own developer documentation. The heaviest use is in software documentation, where coding assistants follow the file to find the right reference page. Assistants with web access can also read it when they're helping someone with a task on your site.
What it isn't, at least today, is a signal that search engines use to rank pages. Google's documentation on AI features in Search says plainly that you don't need to create new machine-readable files or special text files to appear in AI Overviews or AI Mode. None of the main AI search tools has said llms.txt decides whether a site is cited.
So think of llms.txt as a courtesy to AI agents, not a lever for search visibility.




