What Is llms.txt — and Does Your Business Need One?

What exactly is llms.txt?

A markdown file served at yourdomain.com/llms.txt. The convention was proposed in late 2024 as an answer to a real problem: AI engines assemble answers from what they can read, and most websites are hard to read — navigation chrome, scripts, marketing copy that never quite says what the business does. llms.txt is the plain-language version: a short description of the business, a curated list of the pages that matter, each with a one-line summary.

Think of the difference this way: your sitemap tells machines everything that exists on your site. Your llms.txt tells them what matters and what it means. One is an index; the other is a front door.

Does anything actually read it?

Honest answer: adoption by the engines is unofficial and uneven, and anyone who tells you otherwise is selling something. No major AI engine has formally committed to fetching llms.txt the way search engines committed to robots.txt and sitemaps. Some AI tools and agents read it opportunistically; documentation sites have adopted it widely; most business sites haven't.

And the measurement studies back that up. Independent 2026 analyses — including one across roughly 300,000 domains and another across nearly 95,000 AI-cited URLs — found no measurable correlation between having an llms.txt and how often AI engines cite a site. Google has said it won't support the file, and one of its search advocates compared it to the old keywords meta tag. If someone is pitching llms.txt as the thing that gets you cited, that's the data they're not showing you.

So why ship it at all? Four honest reasons. The cost is trivial — an hour, maybe two. It's a hedge — only a small minority of sites have one, so if engines ever formalize support, you're already on record. The exercise itself pays — forcing your business into one page of plain, factual language is exactly the clarity work that improves how engines read the rest of your site. And it's a correctness surface — a single machine-readable statement of your business that your own tools and agents can check; ours is monitored by our audit tooling, which is how we caught a stale price in it before the engines could quote one. What you should not do is buy it as a citation play.

And critically — llms.txt is the speculative layer of a three-file stack. The other two layers work mechanically, today.

What are the three files in the stack?

1. robots.txt — let the AI crawlers in. This one is not speculative. If GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and bingbot are blocked — and plenty of sites block them by accident, via an overzealous CDN setting or a copy-pasted robots file — the engines simply cannot read you, and you will not be recommended. Explicitly allowing them is the single most mechanical fix in AI visibility. (Worth knowing: Google's AI Overviews ride regular Googlebot, and ChatGPT's search leans on Bing's index, so bingbot matters more than it used to.)
2. llms.txt — the curated front door. The machine-readable summary described above: who you are, what you offer, your key pages with one-line descriptions, your service area.
3. agents.md — house rules for agents. The newest layer, aimed at AI agents acting on a user's behalf — comparing providers, drafting a booking, summarizing options. It states what's canonical (your site over third-party listings), where current pricing lives, the one right way to contact or book, and what should never be claimed about you.

What breaks it? The mistakes that make the files worthless

• A redirect. The files must live at the domain root and answer with a plain 200. A 301 from /llms.txt — common when a CMS routes everything through a path handler — breaks crawler access. Test it: curl -I yourdomain.com/llms.txt should say 200, not 301.
• Claims with no page behind them. llms.txt curates what exists; it never introduces new facts. A service listed in the file but absent from the site reads as exactly the inconsistency that makes engines skip you.
• Stated prices. Answer engines quote this file verbatim and don't re-check it often. A price that later changes becomes a machine-readable wrong answer with your name on it. Link your pricing page; let one URL be the single source of truth.
• Set-and-forget. A stale llms.txt is worse than none — it's the confident, quotable version of who you used to be. Update it whenever your offerings change; skim it quarterly regardless.

How do I set this up?

Download the starter kit at the top of this page. It contains all three templates with fill-in-the-brackets guidance, the AI-crawler allowlist block, and a verification checklist — and it's written so you can hand the whole file to an AI agent (Claude Code or similar) with one instruction: "Read llms-txt-starter.md and build our llms.txt stack. Base it on our real pages — crawl the site first, and interview me for anything you can't verify." The agent crawls your site, drafts the files from what's actually true, and runs the verification itself.

If you'd rather see a production example first, ours is live: tkcgroup.co/llms.txt. Note what it does — plain description, curated links, pricing deferred to one canonical page — because the shape matters more than the length.

Frequently asked questions