llms.txt: What It Is and Whether It Actually Helps AI Visibility

What llms.txt actually is

llms.txt is a single markdown file you place at the root of your domain, at yoursite.com/llms.txt. It summarizes what your site is and links to the pages you consider most important, so a large language model can orient itself without crawling everything. Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed the convention in 2024 as a way to save tokens and cut guesswork when an AI reads a site.

Two clarifications matter, because most of the confusion starts here. Despite the name, llms.txt is nothing like robots.txt. It controls nothing and blocks nothing. It is a suggestion, not a rule. It is also separate from the practice of publishing markdown copies of your pages, which is a different tactic with its own tradeoffs. The "get cited in AI search" framing came later, added by the SEO industry as adoption spread, on the bet that AI platforms would eventually reward sites that published one. That bet is the thing worth testing.

The data: most llms.txt files are never read

In May 2026, Ahrefs analyzed 137,000 domains using their server logs and bot analytics, the largest look at llms.txt in the wild so far. The headline number is hard to argue with. 28% of those domains publish an llms.txt file, and 97% of those files received zero requests in the study month. Nothing fetched them at all.

Of the 3% that saw any traffic, 96% of the requests came from bots, and most of those bots were not AI tools. The bots that answer questions inside ChatGPT, Perplexity, and AI Overviews, the retrieval bots that would actually decide your AI visibility, accounted for about 1.1% of requests. Ahrefs is careful to note that a fetch is not the same as a read, so every one of those figures is a ceiling rather than a floor. Real use sits at or below what the logs show.

One finding cuts through the whole debate. No AI bot ever requested an llms.txt file that did not exist. They do not go looking for one. Publishing a file does not put you on a list or a radar, because there is no list. If a file is neither linked nor requested, it may as well not be there.

The mirror image is just as telling. When Ahrefs looked at requests for llms.txt files that returned a 404, the traffic flipped: 98% of it was human, and the AI-bot share was zero. Another 12% of the real llms.txt traffic came not from anything answering a user's question but from the industry studying itself, the GEO and AEO tools and checkers that audit these files. A supply chain grew up around llms.txt before its readers did.

Google says its own search ignores the file

If you want a second opinion, Google has been unusually direct. Its guidance for AI search now includes a mythbusting note stating that you do not need to create machine-readable files, AI text files, or markdown to appear in Google Search or its AI experiences, because Google Search does not use them. Maintaining an llms.txt is fine, the guidance says, but it will not help or hurt your rankings, because Search ignores it.

John Mueller of Google called llms.txt a "temporary crutch" for AI coding tools and described its use for search discovery as speculative. He compared a self-declared summary of your own site to the old meta keywords tag, which search engines stopped trusting once every site stuffed it. His logic is simple. A file where you tell an AI which of your pages are best and which products to recommend cannot, by design, help that AI decide between you and a competitor making the same claim. Gary Illyes has said Google has no plans to support the file. For the job people hope llms.txt will do, Google points instead at a different emerging standard, WebMCP, which targets what an agent does once it is already interacting with your site.

The one job llms.txt does today

There is a real use case, and it is narrow. The AI bots that fetch llms.txt most often are coding agents. In the Ahrefs data, GPTBot and Claude-Code sit at the top of the AI category, ahead of every AI search and assistant bot. These are agents parsing developer documentation, using the index to find the right reference page without reading an entire docs site.

That is why the pattern holds where you would expect it to. Anthropic and OpenAI both publish an llms.txt for their own developer docs, not for their marketing sites. The file earns its keep for an agent that is already on your site and needs to move through it, not for an engine deciding whether to mention you in an answer. Mueller framed it the same way. An llms.txt might help once someone, or something, is already on your site, the way a store directory helps a shopper who has already walked in. If your buyers point coding agents at your product, or agents act on your site to complete a task, the file has a plausible reason to exist. For AI search discovery, it does not.

There is a forward-looking case worth naming honestly. Google has said the future of search is agentic, and if agents end up mediating AI answers rather than retrieval bots fetching pages directly, a file written for agents could start to matter through that layer. That is a reason to keep a clean llms.txt when your platform publishes one for free. It is not a reason to treat the file as a growth lever today. Agents fetch it when they are pointed at it, not on their own, so an unlinked file still waits for a reader that has no reason to arrive.

Cheap is not the same as valuable

Here is where operators get it wrong. Publishing an llms.txt is close to free. It takes about twenty minutes, and website builders are starting to generate one for you. Wix already does. Framer and others are moving that way. Within a year, having an llms.txt will likely be a CMS default, the way a sitemap is today. Near-zero cost, near-zero downside. So if it is free, take it.

The mistake is treating a cheap task as a valuable one and letting it absorb the small amount of attention you have for AI search. I spent seven years running marketing for an Inc. 5000 company from startup to exit, and the same pattern repeats in every channel. Teams do the visible, easy thing and call it strategy. An llms.txt is the AI-search version of that. It looks like progress on your AEO to-do list while moving nothing a buyer or an AI engine responds to.

This is the same trap as schema markup, one step further. In a separate breakdown of schema markup for AI, the honest read was that structured data gets read but produces no measurable citation lift. llms.txt does not even clear that bar. The retrieval bots barely fetch it. Schema is a small lever. For AI visibility today, llms.txt is not a lever at all.

Where the AEO hour actually pays

If you have one hour for AI search this week, two things have evidence behind them.

First, be retrievable. Google's own guidance is that showing up in AI search requires no special files, only that your pages are indexed and your content is legible in self-contained passages an engine can lift. That is ordinary technical SEO and clear writing, not a new file format. Our SEO and AEO work starts here, because nothing downstream matters if the engine cannot read and trust the page.

Second, earn off-site mentions. Ahrefs studied 75,000 brands and found that branded web mentions correlated with AI visibility at 0.664, roughly three times the 0.218 for backlinks. The signals that predict whether an AI names you live mostly off your own site, in what other people say about you. A file you write about yourself cannot compete with that, which is the deeper reason Mueller distrusts self-declared inputs. Our AEO checklist and our guide to ranking in AI search both order the work by what actually correlates with getting cited.

One caution if you do publish. A stale or auto-generated llms.txt is a small security surface. Because agents are built to trust the file, bad actors have started probing llms.txt for prompt injection. Treat it like code: version it, restrict who can edit it, keep it to plain links and descriptions, and review anything your platform generates for you.

The bottom line

llms.txt is scaffolding for a standard that may or may not arrive. The tutorials oversell it as an AI-visibility win; the logs say it is closer to a sitemap nobody has requested yet. Publish it if your platform hands it to you, and keep the file clean. Then check your own server logs before you spend another minute on it. If you want to track AI visibility properly while you are in there, measure it as a prompt set, not a rank. The base rate is a 97% chance nothing reads the file.

Then put the real work where the evidence points: an indexable, well-structured site and a brand that gets mentioned in places you do not own. Those two jobs belong to the same connected system, run on one number, booked sales conversations, which is how we build the Demand Engine rather than a pile of AEO tasks. If you are not sure which of those is your actual constraint, that is the first thing worth finding out.