Is llms.txt worth adding in 2026?

If you publish developer documentation or an API, yes, because the mechanism is direct retrieval: a developer or their coding assistant fetches the file on purpose, and a clean Markdown index of your current documentation is better source material than a heavy documentation app. For everyone else it is optional and low priority. There is no published commitment from the major answer engines to use it, so treat it as documentation ergonomics rather than visibility work.

Does Google use llms.txt?

No. Google has publicly stated that it does not support the llms.txt format. That is a published statement from the engine itself, not an inference from our own testing.

Is llms.txt the same as robots.txt?

No. robots.txt is a long-standing permission file that well-behaved crawlers read before requesting your pages, and it can allow or disallow paths. llms.txt carries no permission semantics at all. It cannot allow or block anything. It is a curated Markdown index that suggests what a language model should read, and its effect depends entirely on something choosing to fetch and honour it.

Does AI Site View score llms.txt?

No. We detect llms.txt and report whether it is present, and it moves the score in neither direction. Presence earns no points and absence costs none. We do not award points for technical fashions without sufficient justification. We will score it the day there is sufficient reason to believe it matters.

Can adding llms.txt hurt my website?

A plain Markdown file at your root does no technical harm. The real cost is opportunity cost and false comfort: an llms.txt does nothing for a site that refuses retrieval crawlers at the door, or whose content is not in the HTML the server delivers. Fix those first, because they are measurable and they change what a machine can retrieve.

Learn

Does llms.txt actually work?

Mostly, no. Not yet, and not in the way it is usually sold. It is a thoughtful proposal with one narrow place where it genuinely earns its keep, and a great deal of advice built on top of it that the evidence does not support. We detect the file. We refuse to give it a single point. Here is the whole reasoning, including what would change our minds.

What llms.txt actually is

llms.txt is a plain Markdown file published at the root of a site, at /llms.txt. It was proposed in 2024 by Jeremy Howard of Answer.AI, and the thinking behind it is sound.

A modern web page is mostly not the content. It is navigation, cookie banners, scripts, tracking, related-article rails and layout. A language model reading that page has to spend limited attention separating the substance from the furniture. llms.txt proposes that you do that separation yourself, once, and publish the result: a short curated index of what matters on your site, in clean Markdown, with a one-line description of each entry.

In practice a file is a title, a blockquote summarising what the site is, then a few grouped lists of links with descriptions. Some sites also publish llms-full.txt, which inlines the full text of the important pages rather than linking to them. Both are conventions rather than a ratified standard.

The one-sentence version

A sitemap.xml tells a machine which addresses exist. An llms.txt tells a machine which of them you think are worth reading, and in what order. The first is addressing. The second is editorial advice, and advice only works if somebody takes it.

llms.txt versus robots.txt: not the same job

These two get compared constantly because both are text files at the root of a domain. That is where the similarity ends, and the difference is the whole argument.

robots.txt

Thirty years old. Read by well-behaved crawlers before they request your pages. It states permission: which paths a named crawler may or may not fetch. Every major crawler operator documents the token it obeys. Getting it wrong has an immediate, observable consequence, because a disallowed path stops being fetched.

llms.txt

Two years old. Carries no permission semantics whatsoever. It cannot allow anything and it cannot block anything. It is a suggestion about what to read, and its entire effect depends on something choosing to fetch it and act on it. No major answer engine has publicly committed to doing so.

So the honest comparison is not "which file should I use". Only one of them is a control. If you want to understand the layer that genuinely decides what reaches an engine, read how AI crawlers actually work, because stated permission and observed behaviour are two different things and they disagree more often than people expect.

What the published evidence says

We have not run a controlled study of llms.txt, and we are not going to imply we have. What follows is other people's published work, described as what it is.

Google has publicly stated that it does not support the format. That is a statement from the engine itself, which is the strongest kind of evidence available on a question like this, and it points one way.
Independent measurement found effectively no traffic attributable to it. An Ahrefs study of roughly 137,000 sites found that about 97 percent received no traffic from their llms.txt. That is measurement at a scale no individual site owner can reproduce.
No major model provider has publicly committed to it as a production ranking or answer signal. Not a refusal in every case, but not a commitment either, and a recommendation should not be built on a silence.

Being fair to the format. None of this proves llms.txt can never work. It shows that today the case for it is unproven, which is a different claim and a weaker one. Conventions do get adopted. If an engine that answers questions publicly commits to consuming it, and independent measurement shows an effect, our position changes and so does our score.

Where llms.txt genuinely earns its place

There is a real use case with a real mechanism, and it deserves stating plainly rather than being buried under the scepticism.

Developer documentation sites and API products are the strong case. Stripe, Vercel, Cloudflare and Anthropic are among the companies that ship one. Their audience is developers building an integration with an AI coding assistant open beside them, and that changes the mechanism completely.

Nothing has to have adopted a standard for this to work. The developer, or the tool acting for them, fetches the file deliberately, because it was handed the address. That is direct retrieval, not ranking. And a curated Markdown index of your current documentation is better source material than a documentation app that assembles itself in the browser, and far better than whatever version of your API a model happened to learn months ago. Fewer invented parameters, fewer deprecated endpoints, less time in your support queue.

If that is your product, ship one. Just be clear with yourself about what you are buying: documentation ergonomics at the point of use, not visibility in an answer engine.

What AI Site View does about it

We detect llms.txt and report whether it is present. It moves your score in neither direction. Presence earns no points. Absence costs none.

From our methodology llms.txt is detected and reported, never scored. Google has confirmed no support for it, and an Ahrefs study of 137,000 sites found 97 percent received no traffic from it. Awarding points for it would be selling a superstition.

Doing the opposite would have been easy. A checklist item costs nothing to add, and a tool that checks more things looks more thorough. We could have folded llms.txt into the score, handed most sites an instant deduction, and sold the fix. That is exactly the kind of recommendation that spends an afternoon of somebody's week on a file that may do nothing, and it is why so much AI visibility advice is worth so little.

The rule is on our methodology page and it is not specific to this format: we do not award points for technical fashions without sufficient justification. Every point in the score is attached to something we observed. We will score llms.txt the day there is sufficient reason to believe it matters, and the methodology version will change when we do.

How to judge the next one for yourself

llms.txt will not be the last convention promoted as essential for AI visibility. There will be a new file, a new meta tag, a new schema type, and a wave of advice saying that sites without it are invisible. Three questions separate a signal that matters from a fashion, and you can apply them yourself in about ten minutes.

1. Has the engine that would consume it publicly committed to it? Not a blog post by an enthusiast. A statement from the company whose product would have to read the thing. If the consumer has said nothing, or has said no, you are looking at a proposal, not a signal.
2. Is there independent measurement of an effect? Someone who is not selling the fix, measuring across enough sites that a result means something. Case studies of one site prove nothing, because sites change several things at once and then credit the newest.
3. Does it change what a machine can retrieve or establish? This is the deepest test and the one that survives every trend. Adding a file that nothing fetches changes none of the bytes an engine receives. Removing a block, or moving your prices out of an image and into the HTML your server delivers, changes them immediately.

How to read the answers. Three out of three, do it now. Only the third, treat it as cheap insurance and get to it once the important work is finished. None of the three, wait. The cost of waiting on a genuine signal is small. The cost of chasing every fashion is the whole afternoon, every time.

What demonstrably does matter

Three things pass all three questions comfortably, and all three are measurable today.

Whether retrieval crawlers are refused at the door

A firewall rule, a bot-protection setting or a rate limiter can answer 403 to a named crawler while your homepage serves perfectly to every human visitor, and nothing in your analytics will mention it. This is not theoretical. It is the failure that made us build the product. We test it by sending real requests with each crawler's published user agent and reporting the verdict, which is either that no access barrier was detected or exactly what came back instead.

Whether your content is in the server-delivered HTML

Rendering behaviour differs from crawler to crawler and changes over time, and many automated crawlers rely heavily on server-delivered HTML. So the durable check is whether your important content exists in the HTML your server returns, before any framework has booted. A file full of curated links cannot rescue a page whose text only appears after hydration.

Whether your business facts are establishable

What you are called, what you do, who it is for, what it costs, where you are, how to reach you. Stated explicitly in structured data or in plain HTML, so a machine can extract them rather than infer them. Our score puts 60 points on Technical Access and 40 on Machine Readability, and every one of those points is attached to something observed on the page rather than to a convention somebody hopes will be adopted.

So, should you add an llms.txt?

If you publish developer documentation or sell an API, yes. The retrieval mechanism is real and the file is cheap to maintain.

For everyone else it is fine to add, and it will cost you an hour. Just put it last, and do not mistake it for the work. An llms.txt on a site that turns retrieval crawlers away at the door, or whose content never reaches the HTML the server sends, is a note pinned to a locked door.

Check the things that are measurable

Free, no signup. Real requests sent with each crawler's published user agent, what your server-delivered HTML states about your business, and every finding traceable to the evidence it came from. We will tell you whether you have an llms.txt. It will not change your score.

Check my site

The evidence described above is published third-party statements and independent measurement, not our own experiments. Everything stated here about our own method is documented on the methodology page. Positions on emerging formats are reviewed as evidence appears.