llms.txt in Practice: How to Create, Deploy, and Honestly Evaluate the File

Dominik Breitbach · Founder & Lead SEO Strategist at taismo

Dominik Breitbach is the founder of taismo GmbH, an SEO and GEO agency from Munich, and specializes in structured data and AI visibility (GEO). Since 2019 he and the taismo team have supported companies across the full breadth of search engine optimization.

⏱ Reading time: 16 min🔄 Last updated: 24 August 2026

An llms.txt file is a curated plain-text file in the root directory of a website (https://your-domain.com/llms.txt) that tells AI systems such as ChatGPT, Claude, Gemini, and Perplexity in Markdown syntax who you are and which of your pages matter most. This guide shows you what belongs in the file, how you create it in 5 steps, and how you deploy and maintain it safely. It also shows you how to check your server logs for the AI crawlers that actually fetch the file, and it gives you the counterargument in full: Google publicly compares llms.txt to the keywords meta tag, and we explain why we maintain one anyway.

👉 You want AI systems to cite you as a source instead of your competitors? That is exactly what we work on in our GEO services.

What is an llms.txt file?

An llms.txt file is a briefing document for AI crawlers: A text file served at /llms.txt that lists the most important pages of a website in Markdown syntax, adds one explanatory sentence per page, and condenses the positioning of the brand into a few lines. Jeremy Howard, AI researcher and co-founder of Answer.AI, proposed the format on September 3, 2024; the specification lives at llmstxt.org. His argument runs in one direction: Language models work with limited context windows, and HTML pages full of navigation, ads, and scripts are expensive for them to parse. A lean, curated Markdown file delivers the same information in the format language models read most efficiently.

The proposal aims primarily at inference time, the moment an AI assistant fetches a page to ground an answer it is writing right now. That distinguishes llms.txt from the crawler permissions debate around model training: The file does not block or allow anything. It offers. Whether any system accepts the offer is the open question this guide measures instead of assuming, and the section on log analysis below shows you how.

The compact definition with all building blocks sits in our glossary entry on llms.txt; how the file fits into AI visibility as a whole is covered in the entry on Generative Engine Optimization (GEO). This article deliberately stays on the practical side: Building the file, shipping it, and judging what it can and cannot do.

llms.txt vs. robots.txt vs. sitemap.xml: Three files, three jobs

The llms.txt is regularly confused with the two other machine-facing text files every website carries. The three do not compete; they answer three different questions:

The last column is the honest part of the table and worth repeating: The robots.txt is binding for well-behaved crawlers, the sitemap is an established discovery signal, and the llms.txt is an invitation without a confirmed reader. It replaces neither of the other two files. If you delete your sitemap because you now have an llms.txt, you have traded a working signal for a speculative one.

The llms.txt format: Four building blocks in a fixed order

The specification is short, and that is its strength. A complete llms.txt consists of four building blocks:

1. H1 line: # Brand | Short claim. One line that makes brand and category unambiguous. The H1 is the only element the specification strictly requires.
2. Blockquote briefing: A few lines starting with > that carry the positioning, the core facts (founding year, location, key people with their roles), and verifiable proof such as awards or review scores.
3. Link sections: ## headings such as Services, Case studies, or Locations, each followed by entries in the pattern - [Title](absolute URL): One dense sentence about the benefit and intent of the page.
4. Legal pages: Imprint and privacy policy at the end, as a trust signal.

The specification adds one reserved section name on top: A section headed ## Optional marks links that a reader with a tight context budget may skip. Everything outside that section counts as primary. A minimal, complete file looks like this:

# taismo | SEO and GEO agency from Munich

> Founded in 2019, offices in Munich and Tuttlingen.
> Ranked 5th in the official German SEO Contest 2026.

## Services
- [SEO services](https://taismo.de/en/seo-services/): Ongoing search engine
  optimization across strategy, content, internal linking, and structured data
- [GEO](https://taismo.de/en/geo/): AI visibility in ChatGPT, Perplexity,
  and Google AI Overviews

## Glossary
- [What is GEO?](https://taismo.de/en/what-is/generative-engine-optimization/):
  Definition of Generative Engine Optimization with methods and examples

## Legal
- [Imprint](https://taismo.de/en/imprint/): Legal information and contact

llms-full.txt and Markdown page copies

The proposal describes two optional extensions. The llms-full.txt bundles the complete content of the listed pages into one large Markdown document, so a language model can ingest the whole site in a single fetch. And for individual pages, the proposal suggests serving a clean Markdown copy at the same URL with .md appended. Both extensions make sense for documentation platforms with heavy developer traffic. For a typical business website, the compact llms.txt is the right scope; start there.

What belongs in an llms.txt (and what stays out)

The most important rule fits in three words: Curate, do not dump. An llms.txt that simply copies the sitemap misses its purpose, because the file exists to weight pages, and a list of everything weights nothing. In our own practice as an agency, 30 to 70 entries and a file size of 2 to 7 KB have proven to be the workable range: Money pages and core pages first, then services, industries, or regions, plus a selection of the 5 to 10 strongest glossary definitions as semantic context.

The second rule concerns the description sentences, because they carry the actual value of the file. Every entry needs one sentence with substance. An entry like “Here you find our services” is worthless; an entry like “Ongoing search engine optimization covering strategy, content, internal structure, and structured data, cancellable monthly” gives the language model facts it can work with. Write the sentence for a machine that has never seen your website and will quote you literally.

What stays out follows from the same logic: Paginated archives, tag pages, thin landing page variants, campaign URLs with tracking parameters, and every page you would not want quoted as a primary source. The file is your official self-description. Treat every line as something an AI assistant may repeat to a potential customer.

How to create an llms.txt in 5 steps

This is the workflow we use for every llms.txt, for our own website and for client projects alike:

1. Step 1: Collect the facts. Claim, positioning, locations, key people, and proof come from verified sources only: The imprint, the about page, and the structured data of the website. The llms.txt must tell the same story as the schema markup. If the two contradict each other, the website loses credibility with exactly the systems it is trying to convince.
2. Step 2: Curate the pages. Select from the sitemap the pages that genuinely explain the business: Services, core pages, case studies, the strongest expert content. Everything else stays out, however proud you are of it.
3. Step 3: Write the description sentences. One sentence per entry, carrying the benefit and the distinguishing feature. This is the most labor-intensive step and the most valuable one, because the real translation work happens here: From “this page exists” to “this page means something”.
4. Step 4: Verify every URL. Every link must answer live with HTTP 200. No redirects, no pages blocked by robots.txt, no drafts. Dead links in a document whose entire job is building trust are the worst possible signal.
5. Step 5: Serve it as a plain-text file. UTF-8 encoding, Markdown syntax, reachable at https://your-domain.com/llms.txt, delivered with the text/plain content type. Keep the URLs absolute, keep brand names exactly as they are written, and keep formatting tricks out; the reader is a parser, not a designer.

llms.txt generators and WordPress plugins: A draft, not a final file

Search demand shows that many site owners want a shortcut: “llms.txt generator” is queried about 590 times per month in the US alone, almost as often as informational queries about the file itself. Generators exist in two flavors, and both deliver the same thing: A starting point, not a finished file.

• Online generators crawl your website and output a Markdown list of URLs with descriptions pulled from meta tags. That automates exactly the step that needed no automation (listing URLs) and skips the step that creates the value (curating and writing one dense sentence per page). Treat the output as scaffolding.
• CMS plugins generate the file directly from your content structure. In WordPress, several SEO plugins, Yoast SEO among them, can produce a basic llms.txt automatically. The convenience is real, and so is the limitation: A plugin knows your page titles, not your positioning.

One warning applies to both flavors. If a plugin generates the file and a hand-written physical file sits in the web root, two sources compete for the same URL. One of them silently wins, and sooner or later someone edits the losing one and wonders why nothing changes. Decide on one source and switch the other off; the deployment section below covers this in detail.

Example: The bilingual llms.txt of a German SEO agency

You can inspect a live specimen anytime: Our own file is served at taismo.de/llms.txt. taismo is an SEO and GEO agency from Munich working for clients in German and English, and the file reflects that: It spans roughly 14,000 characters with 72 links, and after the German part a dedicated section headed ## English version opens up the English pages under /en/. Three decisions in that file are deliberate, and all three transfer to any website:

• Proof first: The first link section leads with case studies and verifiable results, each with source and date. Anyone can write a self-description; a language model that is asked to judge trustworthiness needs evidence it can check.
• Glossary as semantic context: A dedicated section lists definitions from our SEO wiki. It shows AI systems which concepts we work with and anchors the brand in its topic field.
• Both languages in one file: Instead of a separate file under /en/, one English section at the end lists the English pages. That keeps maintenance in one place and avoids collisions with the URL routing of the CMS. On WordPress with a multilingual plugin, a physical file inside the language path can even break the language route itself; one file at the root sidesteps the whole problem.

Deploying and maintaining the file: Backup, health check, rollback

Technically there are two ways to ship an llms.txt. Either it sits as a physical file in the web root, uploaded via FTP or SFTP, or a plugin or code snippet serves it as a virtual route. Both work. Only the mixture is dangerous, as described above: Two sources for one URL, one silent winner, one team member editing the wrong file. Pick one path.

Because the file lives in the web root, we treat every update like a small production deployment, with a fixed sequence of 4 steps: Back up the live version, upload, health check (does the URL answer with HTTP 200 and the text/plain content type, and is the content what you uploaded?), and on any anomaly roll back to the backup. That sounds like ceremony for a text file; it takes two minutes and has already saved us from broken deployments.

Maintenance is the part almost everyone underestimates. An llms.txt is not a one-off project: Every important new page belongs in it, every deleted page must come out. From our own experience: The llms.txt shows up in no standard link checker that crawls your page inventory, so dead links creep in unnoticed. During a routine audit we found two links to deleted pages in our own file; since then, the llms.txt check is a fixed part of every publish and delete workflow. If you are building a new website, put the file into the stack from day one; in our SEO web design projects it ships together with structured data and clean code as part of the standard equipment.

The 6 most common llms.txt mistakes

We review llms.txt files during audits and onboarding, and the same six mistakes keep coming back. Each one is cheap to avoid and expensive to leave in, because every flaw sits in the one document you wrote specifically to be quoted:

1. Dumping the sitemap. A file with 500 unweighted URLs answers the one question the llms.txt exists for (which pages matter most?) with “all of them”. That is no answer. Cut it down to the pages that explain the business.
2. Relative URLs. Entries like [Services](/services/) break the moment the file is read outside its own domain context, and that is exactly how AI systems read it. Every URL in the file is absolute, including protocol and domain.
3. Descriptions that restate the title. “[Pricing](URL): Our pricing page” adds zero information. The description sentence exists to carry facts the title cannot: What is offered, for whom, with which distinguishing feature.
4. Listing pages the crawler cannot or should not use. Redirected URLs, pages blocked by robots.txt, drafts, and deleted pages all undermine the file. A briefing document that points to closed doors reads like an outdated brochure.
5. Two sources for one URL. A plugin-generated file plus a physical file in the web root, as covered above. One wins silently; the other collects edits that never go live.
6. Set-and-forget. The file rots faster than the website because no standard tool monitors it. Without a fixed maintenance trigger, the llms.txt of 2026 still describes the company of 2024.

The pattern behind all six is the same: The llms.txt fails exactly where it is treated as a checkbox instead of a maintained self-description. If you only remember one thing from this list, make it the first point; curation is the entire product.

Log analysis: Which AI crawlers fetch your llms.txt?

Whether the file gets fetched is not a matter of belief; you can measure it. Every request lands in the access log of your web server. On Apache and nginx the file is typically called access.log; on Plesk servers it sits per domain under logs/access_ssl_log. One command answers the first question of who requests the file and how often.

grep -i "llms.txt" access_ssl_log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

The command filters all requests for the llms.txt, extracts the user agent, and counts the hits per agent. For older, rotated logs, use zgrep instead of grep. The second question matters at least as much: Do AI crawlers visit your website at all? You answer it with the same pattern across the whole log, filtered by the bot names.

grep -Ei "gptbot|oai-searchbot|chatgpt-user|claudebot|perplexitybot|perplexity-user" access_ssl_log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

These are the user agents worth knowing, taken from the official crawler documentation of the providers:

The last row corrects a widespread misreading: Whoever greps the log for “Google-Extended”, finds nothing, and concludes that Google ignores the website for AI purposes has fallen for a phantom. Googlebot crawls as always; Google-Extended exists only as a switch inside the robots.txt. Which of these switches work independently of each other, and why blocking GPTBot does not remove you from ChatGPT search, is broken down in our article on ChatGPT SEO.

Does the llms.txt file help your SEO?

For classic Google rankings, the answer is a clean no: The llms.txt has no confirmed effect on Google search rankings. Google does not parse the file for search, and no ranking system reads it. If your goal is a better position in the classic result list, invest the same hours in content, internal linking, and structured data; every one of those has documented effects, the llms.txt has none.

The realistic case for the file lies one layer further out, in AI-mediated visibility. Three honest statements describe the state of play:

1. The llms.txt is preparation, not optimization. The file positions you for a reading behavior that providers may adopt more broadly. It costs hours, not budgets, which is why the bet is rational despite the uncertainty.
2. The measurable SEO value today is indirect. Writing the file forces a canonical self-description, reconciled with the schema markup and the website. Inconsistencies you find and fix in that process (divergent claims, dead pages, unclear positioning) improve signals that search engines demonstrably use.
3. AI visibility follows content quality, not file presence. Answer engines cite pages because the pages answer well. The llms.txt can point a system at your best pages; it cannot make weak pages citable.

In short, treat the llms.txt as part of your GEO toolkit and not as an SEO tactic. Whoever promises you rankings from a text file has skipped the evidence; the next section shows what the most prominent skeptic says, and why he is partly right.

The counterargument: What Google thinks of llms.txt

An honest guide names the opposing position, and this one is prominently staffed. Google Search Advocate John Mueller stated in a Reddit discussion in April 2025 that he knows of no AI service that uses the llms.txt, and that a look at server logs shows the bots do not even fetch the file. His comparison stings: The llms.txt resembles the keywords meta tag, a self-declaration by the site owner that search engines have ignored for years because it is trivially manipulable. If an AI system has to verify the self-description anyway, it can just as well read the website itself (source: Search Engine Journal, April 17, 2025).

Take this criticism seriously, because three facts inside it are undisputed: The llms.txt is not an official standard but a proposal. No major provider has committed to parsing it. And it is not a confirmed ranking factor, neither in Google search nor in AI answer engines. Anyone selling you an llms.txt as a ranking booster is selling a promise without evidence.

Our assessment and the German SEO Contest 2026 experience

What follows is our assessment as an agency, clearly labeled as opinion and experience, not as a proven ranking factor. We believe the llms.txt, much like the speakable markup in structured data, matters more than official statements admit. Four reasons carry that position:

1. The cost-benefit ratio is extreme. An llms.txt costs a few hours of work, carries zero risk for your rankings, and lies ready in case providers start parsing it more seriously. The keywords meta tag never harmed a website either; it just stopped doing anything at some point.
2. The format matches how the systems work. Answer engines load pages live to ground their answers, and lean Markdown without HTML ballast is the format language models process most efficiently. Telling detail: Anthropic, the company behind Claude, serves an llms.txt for its own developer documentation. The providers use the format exactly where they expect machines as readers.
3. The file forces consistency. Whoever writes a good llms.txt has to state positioning, facts, and proof once, cleanly, and reconcile them with the schema markup and the website. That clarification process pays off regardless of who ends up reading the file.
4. Our contest experience. In the official German SEO Contest 2026 we deployed the complete machine-readable toolkit with our Serponado campaign: Entity building across several domains, structured data, citable content, and an llms.txt on both taismo.de and the campaign domain. The documented result: 4th place at the second scoring date and 5th place in the final ranking among more than 200 participating agencies and SEOs. To be blunt about causality: Nobody can seriously isolate the contribution of the llms.txt to that result, including us. But the overall setup of consistent, machine-readable signals proved itself in one of the hardest competitive environments there is, and the llms.txt was part of that setup from day one.

The limit is just as clear: An llms.txt replaces no substance. The file only points at content; whether AI systems adopt that content is decided by its quality. How you build content that ChatGPT, Perplexity, and AI Overviews actually quote as a source is the subject of our guide on how to get cited by AI. The llms.txt is the map; you still have to build the cities.

Frequently asked questions about llms.txt

What is an llms.txt file?

An llms.txt file is a plain-text file in Markdown syntax in the root directory of a website. It lists the most important pages of a brand for AI systems, adds one explanatory sentence per page, and summarizes the key facts of the brand.

Where does the llms.txt file live and how is it served?

The file lives in the web root and is reachable at https://your-domain.com/llms.txt. It is served as a UTF-8 text file with the text/plain content type.

Is llms.txt an official standard or a ranking factor?

No. The llms.txt is a proposal published by Jeremy Howard of Answer.AI in September 2024. No search engine or AI provider has committed to parsing it, and it is not a confirmed ranking factor.

Do AI crawlers actually read the llms.txt file?

Server logs show only sporadic fetches, and Google states that no major AI provider uses the file. You can verify the behavior for your own website by filtering your access log for llms.txt and for AI user agents such as GPTBot or ClaudeBot.

How do you add an llms.txt file to a WordPress website?

Either upload a hand-written file to the web root via FTP or SFTP, or let an SEO plugin such as Yoast SEO generate one. Use exactly one of the two methods; two sources for the same URL lead to silent conflicts.

How often should you update your llms.txt?

Update the file whenever you publish an important new page and whenever you delete a listed one. Also check all contained links for HTTP 200 regularly, because standard link checkers do not cover the file.

Sources

• Jeremy Howard: “/llms.txt: A proposal to provide information to help LLMs use websites”, Answer.AI, September 3, 2024.
• Answer.AI: “The /llms.txt file”, llms.txt specification, retrieved August 10, 2026.
• Roger Montti: “Google Says LLMs.Txt Comparable To Keywords Meta Tag”, Search Engine Journal, April 17, 2025.
• OpenAI: “OpenAI bots”, OpenAI developer docs, retrieved August 10, 2026.
• Anthropic: “Does Anthropic crawl data from the web?”, Anthropic Support, retrieved August 10, 2026.
• Perplexity: “Perplexity crawlers”, Perplexity docs, retrieved August 10, 2026.
• Google: “Overview of Google crawlers and fetchers”, Google Search Central, retrieved August 10, 2026.