llms.txt, honestly: what it does and what it does not do
July 18, 2026
llms.txt has a strange status. Proposed by Jeremy Howard in September 2024, adopted by a large share of developer documentation sites since, made a default by some hosting platforms, and still, as far as anyone can show publicly, not committed to by a single major AI provider. Google's search people have compared it to the keywords meta tag, which in SEO circles is roughly a death sentence.
So the honest question is not "is llms.txt important". It is "why would I spend any time on a file nobody promises to read".
Here is the case, and it is smaller and more practical than most of what gets written about this file.
The file is a curated index. One markdown page at a fixed address saying: this is what this site is, these are the pages that matter, here is where the actual documentation lives. That is useful to any reader with a small context window and no patience, which describes every agent that will ever visit you. Whether the reader is GPTBot or some internal tool a customer wired up last week does not really matter. When one of them does fetch it, the file either helps or it does not exist.
The cost is twenty minutes. That asymmetry is the whole argument. There are not many things in this space with a twenty minute cost and a plausible upside, and the file cannot hurt you. The keywords meta tag comparison fails in one way: keywords were invisible spam that engines learned to distrust. llms.txt is a public, readable page. If you stuff it with nonsense, humans see the nonsense too.
What it does not do: it does not make you rank, in AI answers or anywhere else. It does not control what models train on. It does not replace your robots rules, and it does not make an unreadable site readable. A site that fails the basic access checks with a beautiful llms.txt is a locked shop with a good sign in the window.
We keep a free generator that builds the file from your pages, and the fuller background is in What is llms.txt. Twenty minutes. Then go fix your bot rules, which matter more.
