Do You Need an llms.txt File? (No, You Don't)

No — almost no website needs an llms.txt file. Google has publicly said it does not use one, and not a single major AI crawler has confirmed that it reads the file. It is an optional community experiment, not a ranking factor and not a requirement for showing up in AI answers.

The idea sounds tidy: drop a clean markdown file at your site root and hand large language models a pre-digested map of your content. But the companies that actually run those models never agreed to read it. This guide explains what an llms.txt file is, why Google's own guidance waves it off, and where your effort belongs if you want to earn citations in ChatGPT, Claude, Perplexity, and Google's AI answers.

What an llms.txt file actually is

The proposal came from Jeremy Howard of Answer.AI in September 2024. The concept is a single markdown file served at your domain root — /llms.txt — that gives a language model a curated summary of your site: a title, a short description, and a hand-picked list of links to your most important pages.

A companion variant, llms-full.txt, goes further and concatenates the full text of your documentation into one file so a model can ingest everything in a single request. Several documentation platforms now generate these automatically, which is why the format spread quickly through developer-tool sites.

The intent is reasonable. Rendered HTML is noisy — navigation, scripts, and markup all burn through a model's limited context window — so a clean markdown digest sounds helpful. The problem is not the intent. The problem is that intent alone does not make crawlers use the file.

What Google says about your llms.txt file

Google's John Mueller has been blunt about it. As widely reported across SEO communities, he has said he is not aware of any AI service that uses llms.txt, and that server logs show the AI crawlers do not even request the file. He compared it directly to the old keywords meta tag — a self-declared signal that search engines learned to ignore because anyone could write anything in it.

That comparison is the whole story. The keywords meta tag let a page describe itself, in its own words, with no verification. It was abused within months and dropped from ranking systems long ago. An llms.txt file follows the same pattern: a site describing itself, with no independent check that the description matches the real pages.

Google Search does not use llms.txt, and Google has not tied it to AI Overviews or AI Mode. So for the single largest source of AI-generated answers on the web, the file does nothing.

Why no AI crawler confirms reading llms.txt

Look at who actually fetches your pages: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, Google-Extended, and PerplexityBot. Every one of them is documented to crawl ordinary HTML. None of their public documentation lists llms.txt as an input for grounding, retrieval, or training.

There are practical reasons they stay quiet on it:

• No verification. A curated summary can claim anything. Trusting it invites the exact manipulation that killed the keywords meta tag.
• Content divergence. If your llms.txt describes pages differently from the HTML people see, that is a form of cloaking — one story for machines, another for humans. Crawlers are built to distrust precisely that.
• Double maintenance. A hand-curated file drifts out of date the moment your real pages change, so its accuracy decays on its own.

Until a major AI provider documents that it reads the file, publishing one is writing a letter that no one has agreed to open.

What actually earns AI citations

AI answers are assembled from sources a model can retrieve, parse, and trust. You influence that with the same fundamentals that have always mattered, sharpened for machine reading:

• Let the crawlers in. Your robots.txt decides whether GPTBot, ClaudeBot, and the rest may fetch you at all. If you want citations, do not block the bots you want citing you.
• Answer in the first lines. Models lift concise, direct answers. Lead with the answer, then support it — the pattern this page itself uses.
• Use clean semantic HTML. Real headings, lists, and tables give a model the structure that llms.txt was trying to fake — inside the page users actually see.
• Add structured data. Schema.org markup (FAQ, Article, Product, Organization) spells out the entities and facts a page contains.
• Earn third-party mentions. Models weight what credible sites say about you far more than what you say about yourself. Citations, reviews, and links from real sources build the authority that gets you quoted.

Every one of these lives in your actual pages and is independently verifiable — which is exactly why crawlers respect them and ignore a self-authored digest.

The strongest AI signal you control: your brand

One asset quietly shapes whether an AI mentions you at all: your brand name. Language models are trained on text where entities are named, and they reproduce the names that appear clearly and consistently. A vague, generic, or hard-to-spell name is easy to garble or omit; a distinctive, brandable one is easy to learn, remember, and cite.

That makes your domain a better investment than any metadata file. If you are naming or renaming a project and want a name an AI can recognize, Atom.com is a strong place to look: it is a curated marketplace of premium, brandable names, and each listing ships with a free AI appraisal, a USPTO trademark check, secure escrow, and a professionally designed logo. Atom also offers domain registration and management, so you can register and run an available name there as well.

For pure at-cost registration, registrars like Porkbun (around $11 a year, free WHOIS privacy) or Cloudflare (near-wholesale) are the cheapest way to grab a plain .com, which still sits around $10–12 in 2026. Atom earns its place when you also want the name itself to be memorable and defensible — the brandable asset that gets your business named in the answer, not merely indexed.

Should you ever bother creating one?

If you run large developer documentation and your platform emits an llms.txt automatically at no cost, leaving it in place does no harm — as long as it stays consistent with your real pages. The risk is spending real hours hand-crafting and maintaining a file with no confirmed reader while your actual content, structure, and authority go unaddressed.

The honest verdict: treat llms.txt as a zero-priority, opportunistic add-on, never a task you schedule. If a tool generates one for free, fine. If building one competes with improving your pages, skip it and improve the pages — that is where the citations come from.

Frequently asked questions

Does Google use llms.txt?

No. Google's John Mueller has said Google does not use it and that AI crawlers do not even request the file. Google compares llms.txt to the long-defunct keywords meta tag — a self-declared signal with no verification.

Do ChatGPT, Claude, or Perplexity read llms.txt?

None of them have documented llms.txt as an input for training, retrieval, or grounding. Their crawlers — GPTBot, ClaudeBot, PerplexityBot and others — fetch standard HTML pages, not a curated markdown file.

Will an llms.txt file help me appear in AI Overviews or rank better?

There is no confirmed effect. Google has not tied llms.txt to AI Overviews, AI Mode, or classic ranking. Crawlable, clearly answered, well-structured pages and third-party authority are what drive AI visibility.

Is it harmful to publish an llms.txt file?

Not inherently, if it stays consistent with your live pages. The real risks are wasted maintenance effort on a file no one confirms reading, and cloaking-like divergence if the file describes your site differently from the HTML users see.

What should I do instead to get cited by AI?

Allow the AI crawlers in robots.txt, answer questions directly at the top of each page, use clean semantic HTML plus schema.org markup, earn credible third-party mentions, and invest in a memorable, brandable name — sources like Atom.com pair premium names with a free AI appraisal and trademark check.