okaihub

llms.txt Explained: What It Is and Whether It Actually Works

The llms.txt convention, what belongs in it, what it does and does not do, and how to publish one — from a site that ships both variants.

Published 2026-09-26 · 4 min read

llms.txt is a proposed convention for giving a large language model a small, structured summary of a website: what the site is, and where its most important content lives. It is a plain markdown file at the root, alongside robots.txt and the sitemap.

It is also genuinely uncertain. The honest position is that it is cheap to publish, plausibly useful, and not something to build a strategy around — which is a different assessment from either of the two you will usually read.

What it is

The convention, as described at llmstxt.org, has two files:

FileContentsSize
/llms.txtA one-paragraph description of the site, then grouped links to its important pages with a line of context eachSmall — a few KB, intended to fit comfortably in a model's context window
/llms-full.txtThe concatenated full text of the important pagesLarge — often megabytes, sometimes capped or split by section
The split exists so a system can decide how much to fetch. A model answering a general question needs the index; one answering a specific question may want the full text.

The format is deliberately simple: markdown headings for grouping, bullet lists of links, each with a short description. No schema, no required fields beyond the description.

What it is for

The stated problem is that a model with limited context cannot usefully ingest a whole website, and that HTML is noisy — navigation, scripts, styling and boilerplate consume most of a page's bytes. llms.txt offers a pre-digested alternative: the important content, in plain text, in a known location.

In practice it serves three functions, in decreasing order of how well established they are:

  • A machine-readable table of contents. Useful to any system that wants to know what a site covers without crawling it. This works regardless of whether any particular assistant reads the file.
  • A clean text source for context. Where a system does fetch it, the content is easier to use than scraped HTML.
  • A signal of intent. Publishing one says the site wants to be readable by models, which matters because most sites actively block them.

Whether it actually works

This is the part usually skipped in articles about it, and the reason to be careful:

ClaimStatus
Major assistants fetch llms.txt when answeringNot established. No provider publishes this as a documented behaviour, and behaviours change without notice
It improves your chance of being citedPlausible but unproven. Retrieval depends mainly on crawler permission and page quality
It is a ranking factorNo. It is not a search ranking signal and nothing suggests it will become one
It costs anything meaningful to publishNo. Generated at build time from content you already have
Adoption is high enough to be table stakesNo. Adoption is still low, which is the entire argument for having one

What to put in it

  1. 1

    One paragraph describing the site

    What it is, what it covers, and what makes its content trustworthy. This is the part most likely to be read, so it should stand alone. Derive counts from real data rather than hardcoding them — a stale number in a file written for machines to trust is worse than no number.

  2. 2

    Grouped links to the important pages

    Grouped by what they are for, not by your navigation structure. Each link gets one line of context. Keep the total small enough to fit comfortably in a context window.

  3. 3

    A link to the full version

    So a system that wants depth can find it without crawling.

  4. 4

    Nothing that requires authentication

    A model cannot log in. Links behind auth are noise in this file.

Implementation notes

  • Generate it at build time from the same data that renders your pages. A hand-maintained llms.txt goes stale, and a stale one is worse than none because it misinforms.
  • Exclude pages you have marked noindex. Listing a page in a machine-facing index while telling crawlers not to index it is a contradiction, and it is the same class of error as listing noindex URLs in a sitemap.
  • Cap the size of llms-full.txt. Concatenating a large site produces many megabytes. Cap it, or split by section with per-section files linked from the index.
  • Serve it as plain text with the right content type, and cache it — it changes only when your content does.
  • Link it from your footer. Machine-facing surfaces should also be human-discoverable; it makes the file verifiable and it signals intent to anyone evaluating the site.

The bigger lever underneath

If you do one thing from this article it should not be llms.txt. It should be robots.txt, because that is the difference between being retrievable and not being retrievable at all — and a site that blocks the retrieval crawlers will never be cited no matter how good its llms.txt is.

The order of value is: permit the crawlers, write definition-first pages, use real structure for facts, keep entity naming consistent — and then publish llms.txt as a cheap extra that might help and cannot hurt.

llms.txt is a bet on a convention that may or may not be adopted. The bet costs an afternoon, so take it — but do not confuse it with the work that determines whether a model can find you at all.

More in SEO & GEO