What is llms.txt?

llms.txt is a standardized Markdown file in the web root of a website (https://domain.com/llms.txt) that gives large language models and AI crawlers a curated view of a site’s most important content and its structure. The purpose of llms.txt is to let language models such as ChatGPT, Claude or Perplexity understand a website correctly and use its central pages reliably. The format was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI. That makes llms.txt the AI counterpart to the classic sitemap.

The file does not address human visitors, it addresses machines. At one single, predictable address it bundles which content of a brand matters most and how a language model should classify it. That makes llms.txt one of the technical building blocks of Generative Engine Optimization (GEO), the optimization for AI search systems.

The term at a glance

Which problem does llms.txt solve?

llms.txt solves the context problem of language models: they cannot process a website as completely as a human can. The reason lies in two technical limits. First, LLMs have a limited context window and cannot read an arbitrary number of pages at once. Second, typical HTML carries a lot of extra material that is irrelevant to the pure information: navigation, ads, scripts, cookie banners and footers.

An llms.txt clears that ballast away. It delivers a curated selection of the most important URLs in clean Markdown, each with one explanatory sentence. The language model does not have to guess which pages are central, it gets the priorities straight from the brand. That raises the chance that an AI reproduces a company’s positioning, services and facts correctly instead of reconstructing them from scattered signals.

For brands that is a direct lever of AI visibility. If you want answer engines to quote your content correctly, you work on large language model optimization (LLMO), and llms.txt is a concrete tool you can put in place quickly.

How is an llms.txt structured?

An llms.txt follows a fixed Markdown structure of four building blocks. The standard from llmstxt.org prescribes exactly one H1, followed by a short summary and any number of thematic sections with link lists.

1. H1 with the name: one single heading carrying the brand or project name. It is the only required element.
2. Blockquote summary: a short paragraph (often as a > quote) describing the brand, its positioning and key facts in one or two sentences.
3. Thematic sections: H2 headings such as “Core pages” or “Services”, followed by link lists in the format [Title](URL): description.
4. Optional section “Optional”: an H2 named Optional for links an AI may skip when context is tight.

A minimal example for taismo would look like this:

# taismo | SEO and GEO agency from Munich

> taismo works with companies in competitive markets that have
> services in need of explanation. Focus: structured data, GEO
> and WordPress development. Based in Munich.

## Core pages
- [SEO services](https://taismo.de/en/seo-services/): Monthly SEO retainer for measurable growth.
- [About us](https://taismo.de/en/about-us/): Team, values and method of the agency.

## SEO glossary (excerpt)
- [What is GEO?](https://taismo.de/en/what-is/generative-engine-optimization/): Definition and levers of AI visibility.

## Optional
- [Imprint](https://taismo.de/en/imprint/): Legal information.

Every entry consists of a linked title and one dense explanatory sentence. Those one-sentence descriptions are the real value: they tell the model not only where a page sits, but also why it is relevant.

llms.txt vs. robots.txt: what is the difference?

llms.txt and robots.txt pursue opposite goals: llms.txt invites AI systems in and curates content, robots.txt steers and restricts crawler access. Both files sit in the web root, but they complement each other instead of replacing each other.

• Purpose: the robots.txt tells crawlers which paths they should not fetch. The llms.txt tells language models which content matters most to them.
• Content: robots.txt carries technical directives (Disallow, Allow, User-agent). llms.txt carries curated links with explanatory prose in Markdown.
• Effect: robots.txt restricts, llms.txt prioritizes and explains.
• Binding force: robots.txt is an established standard that is widely followed. llms.txt is a young proposal and using it is voluntary.

The line against the sitemap matters just as much: a sitemap.xml lists all indexable URLs for search engines. The llms.txt deliberately shows only a subset of the most important pages and adds context for LLMs. If you want the detail, see the term sitemap.

How do you create an llms.txt?

An llms.txt is created in four steps and stored as a file in the web root. The process is deliberately lean, because the file should be curated by hand and not generated automatically.

1. Pick the most important pages: money pages, about us, case studies and a selection of strong guide or glossary pages. Do not copy the whole sitemap, curate it.
2. Write the Markdown: H1 with the brand name, a short summary as a blockquote, then thematic sections with link lists in the format [Title](URL): description.
3. Check every URL: only absolute links that really resolve with HTTP status 200. Redirects and blocked pages do not belong in there.
4. Store the file: as llms.txt in the web root, so that it is served under https://domain.com/llms.txt as text/plain.

The file should stay compact, two to seven kilobytes is a good rule of thumb. taismo runs an llms.txt of its own at taismo.de/llms.txt that follows exactly this structure. If you take machine-readable signals seriously, combine llms.txt with clean JSON-LD: structured data hands machines the facts in a fixed schema, llms.txt hands them the priorities in prose.

Do AI systems use llms.txt yet?

At the moment none of the large AI providers officially evaluates llms.txt across the board. Google has stated several times that Googlebot does not use llms.txt as a ranking or crawling signal. For AI Overviews as well, no evaluation of the file is confirmed.

The picture is still more nuanced than that. The format is already supported widely by tools and documentation platforms, and some AI-adjacent services do read the file when someone queries a page specifically. The effort is small and the risk is zero: an llms.txt does no harm to classic search and is quickly written. For brands betting early on AI visibility, it is a cheap bet on a standard that may establish itself.

To put it in perspective: llms.txt is no substitute for good content, sound technology and structured data, it is an addition. It is one building block of considered SEO services that think classic search engines and AI systems together.

Frequently asked questions about llms.txt

Is llms.txt the same as robots.txt?
No. robots.txt restricts crawler access with technical directives, llms.txt curates and explains the most important content for language models. Both sit in the web root but pursue opposite goals.

Does every website need an llms.txt?
No, it is voluntary. For brands with a lot of content that want to appear correctly in AI systems, the small effort still pays off, because the file prioritizes their own positioning in a machine-readable way.

Does llms.txt improve my Google ranking?
No. By Google’s own statement, llms.txt is not used as a ranking signal. The benefit lies in AI visibility, in language models classifying and reproducing content correctly.

Where does the llms.txt have to sit?
In the web root of the domain, reachable at https://domain.com/llms.txt. Only at that predictable address do crawlers find the file reliably.