Does llms.txt Actually Work? What the Data Shows

Short version, because that’s the whole point of the file we’re discussing: llms.txt is a proposed standard, not an adopted one, and there is no public evidence that any major AI engine reads it to decide what to cite. We shipped one on this site to find out for ourselves. Over the measurement window it was fetched twice — once by a browser, once by a Telegram link-preview bot — and by zero named AI crawlers. That’s a sample of one small site, but it lines up with everything else known publicly. This post lays out what the file is, who’s pushing it, why Google went out of its way to dismiss it, and what the data actually supports doing instead.

What llms.txt is supposed to do

The idea, proposed by Jeremy Howard’s Answer.AI in September 2024, is a sibling to robots.txt and sitemap.xml. You place a Markdown file at yourdomain.com/llms.txt that gives a large language model a clean, curated map of your site: a short description of what the site is, then a list of your most important pages with one-line summaries, optionally split into “essential” and “optional” sections. The pitch is that when an AI model comes to your site, it faces a wall of navigation, ads, and boilerplate; llms.txt hands it the signal without the noise, so the model spends its limited context on what matters.

It’s a genuinely reasonable idea. Markdown is cheap for a model to parse, curation is something a publisher can do better than a scraper, and the format is trivially easy to write — this site’s own file is a few dozen lines. The problem isn’t the concept. The problem is that a file only works if something on the other end reads it, and that’s the part nobody has been able to demonstrate.

Who backs it — and who doesn’t

The adoption story splits cleanly. On the “for” side: Answer.AI proposed it, a directory of hundreds of sites now list one, and several developer-tool companies (documentation platforms especially) ship llms.txt by default because their audience is exactly the developers most likely to point an AI agent at their docs. Tooling sprang up to generate the files automatically. In GEO-consultant circles, “you’re missing your llms.txt” became a standard audit line item.

On the “against” side sits the group that actually operates the crawlers. As of mid-2026, none of OpenAI, Anthropic, Google, or Perplexity has publicly committed to reading llms.txt as a ranking or retrieval signal. Google went further than silence. In its May 2026 AI optimization guide, it stated plainly that “you don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in generative AI search” — a direct, if unnamed, shrug at the entire llms.txt premise. When the company that runs the largest AI-search surface tells you the file is unnecessary, that’s not a neutral data point.

What happened when we shipped one

Talking about this in the abstract is cheap, so we ran the test. This site ships an llms.txt (and a fuller Open Knowledge Format bundle alongside it), and the OKF bundle is instrumented — every fetch is logged to a key-value store with the requesting user agent, so we can see exactly who reads it rather than guessing.

The result, over the logged window: two fetches total. One was a normal desktop browser (a human, or us, opening the file). The other was TelegramBot, the link-preview crawler that fires when a URL gets pasted into a chat. Named AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — accounted for zero. This is a small site with modest traffic, so it’s not proof that llms.txt is inert everywhere; a documentation site with a developer audience pointing agents at it might see real fetches. But it’s a concrete, first-hand data point, and it points the same direction as the public silence from the crawler operators: the file is being published far more than it’s being read.

That mirrors the broader finding from running our full GEO playbook on this site — the citability plumbing shipped cleanly, but the machine-readable extras haven’t yet produced measurable pickup, while the citations we do earn come from being genuinely differentiated on a topic, not from a special file.

Why the file probably underperforms the pitch

There are three structural reasons llms.txt hasn’t become the signal it was pitched as, and understanding them tells you where to spend effort instead.

First, the retrieval engines already have your content. When Perplexity or ChatGPT answers a live query, it runs a search, fetches the pages that rank, and reads them. It doesn’t need a curated map to find your best page — the same ranking systems that surface you in traditional search surface you to the AI’s retriever. A separate file is redundant to a pipeline that already crawls the open web.

Second, an unverified self-description is a spam vector. A file where a site describes its own most important content, in its own words, with no external corroboration, is exactly the kind of signal a search-quality team distrusts on principle — because the moment it counts, it gets gamed. That’s the same reason meta keywords died. Google’s public position (“just keep doing SEO”) is partly self-interest, but this part is genuinely sound.

Third, there’s no adoption flywheel. A standard like this only matters if the major consumers commit to it, and none have. Without a crawler publicly reading llms.txt, publishers ship it as insurance, tooling generates it, and the file accumulates as an artifact nobody consumes — adoption without consumption.

What to actually do

The honest recommendation is not “llms.txt is a scam.” It’s “llms.txt is cheap insurance with no demonstrated payoff, so treat it accordingly.”

• Ship one if it’s free. If your CMS or docs platform generates an llms.txt automatically, leave it on. It costs nothing, it’s not harmful, and if a crawler ever does start honoring it you’re already covered. Just don’t expect it to move anything today.
• Don’t pay for it, and don’t let an audit scare you. If a GEO audit flags a missing llms.txt as a problem, discount that finding. Google explicitly said the file is unnecessary; a consultant billing to add one is selling insurance against a risk that hasn’t materialized.
• Spend the effort on retrieval instead. The things that demonstrably get you cited are the same things that rank you: be crawlable and fast, state the answer plainly at the top of each section, structure content into genuinely extractable units, and build a consistent, corroborated entity so a model can resolve and trust who you are. That’s the whole playbook for getting cited, and none of it depends on a special file.

My take

llms.txt is a good idea that’s waiting on permission it may never get. The concept — hand a model a clean map instead of making it wade through your chrome — is sound, and if OpenAI or Google announced tomorrow that they honor it, this post would age into a footnote. But standards don’t win on elegance; they win on adoption by the parties that matter, and every crawler operator has so far declined. Until that changes, llms.txt belongs in the “ship it if it’s free, ignore it otherwise” bucket, and any effort you were about to spend on it is better aimed at the retrieval fundamentals that actually decide whether a model cites you. We keep our file live and instrumented precisely so we’ll notice the day that changes — and we’ll report it here when it does.

For the wider picture on how models pick and quote sources, see what LLM SEO actually is and the rest of the AI Search · GEO channel.

Frequently asked questions

What is llms.txt?

llms.txt is a proposed standard — a Markdown file placed at the root of your domain (yourdomain.com/llms.txt) that gives AI models a curated summary of your site and a list of its most important pages with short descriptions. It was proposed by Answer.AI in September 2024 as an AI-era sibling to robots.txt and sitemap.xml.

Do ChatGPT, Perplexity, or Google actually read llms.txt?

There is no public confirmation that any major AI engine uses llms.txt as a ranking or retrieval signal, and Google’s May 2026 guidance explicitly says the file is unnecessary. When we shipped an instrumented file on this site, named AI crawlers fetched it zero times over the logged window.

Should I add an llms.txt file to my site?

If your platform generates one automatically, leave it on — it’s cheap and harmless. But don’t pay to have one built and don’t treat a missing llms.txt as an SEO problem. Spend the effort on retrieval fundamentals — crawlability, plainly-stated answers, extractable structure, and a corroborated entity — which demonstrably influence whether a model cites you.

Is llms.txt the same as an XML sitemap?

No. A sitemap lists your URLs for traditional search crawlers and is actively consumed by Google and Bing. llms.txt is a Markdown summary aimed at AI models, and unlike sitemaps it has no committed consumers among the major AI engines yet.