llms.txt is a plain Markdown file, usually at the root of a website, that gives AI tools a short, curated guide to the site’s most important pages. It is a community proposal. Google, OpenAI and Anthropic do not document their crawlers relying on it.
Clients ask about it because SEO plugins now offer a setting for it, and because AI vendors publish their own. This piece walks through what the file contains, how the current version of the proposal works, what Google, OpenAI and Anthropic say on their own pages, and when it is worth adding to a nonprofit, association or university site.
The short version: llms.txt is cheap to add and harmless when done well. It helps most on documentation-heavy sites that people query through AI coding and research tools. Google says you do not need it to appear in AI Overviews or AI Mode. If you have limited hours, fix crawl access, page text and internal links first, then add the file if your CMS makes it easy.
Where llms.txt came from
The proposal lives at llmstxt.org. According to that page, Jeremy Howard proposed it on September 3, 2024. The page now carries the title "The /llms.txt file, v2" and shows a last-modified date of August 10, 2026.
The problem it targets is simple. Web pages are built for people, with navigation, scripts, ads and layout code around the text. An AI tool that reads a page during a chat has a limited context window. Turning a full HTML page into clean text is slow and error-prone. A short Markdown file can point the tool to the pages that matter and to clean text versions of them.
The proposal states its own expectation that the file will "mainly be useful for inference", meaning when a tool reads your site to answer a question, rather than for training models. It adds that training runs could use the information too.
What goes in an llms.txt file
The format is plain Markdown in a fixed order. According to llmstxt.org, the parts are:
| Part | Required? | What it holds |
|---|---|---|
| H1 heading | Yes, the only required part | The name of the site or project |
| Blockquote | No | A short summary of what the site is |
| Detail paragraphs or lists | No | Extra context, with no headings |
| H2 “file lists” | No | Markdown links, each with optional notes after a colon |
| “Optional” H2 | No | Secondary links a tool can skip when it needs a shorter context |
Here is a small example for a made-up arts nonprofit:
# Example Arts Council
> Example Arts Council funds local artists and runs free arts programs for schools.
Grant deadlines change each year. The grants page always has the current dates.
## Programs
- [Artist grants](https://example.org/grants.md): who qualifies and how to apply
- [School programs](https://example.org/schools.md): free workshops for K-12 classes
## About
- [Mission and staff](https://example.org/about.md)
- [Contact](https://example.org/contact.md)
## Optional
- [Annual reports](https://example.org/reports.md)
Keep it short. The value of the file is the editing. A list of every URL on the site is a sitemap, and you already have one of those.
How version 2 of the llms.txt proposal works
The current proposal adds a few rules that older guides miss.
Location. The file can sit at the site root or at any path within it. A file at /docs/llms.txt covers pages under /docs/. When more than one file applies, the proposal says tools should use the most specific one. That helps large association sites where the member knowledge base and the public site have different owners.
Markdown versions of pages. The proposal asks sites to offer a clean Markdown version of useful pages at the same URL with .md added, such as page.html.md or page.md. A URL with no file name uses index.html.md or index.md.
Discovery. A page can point to its Markdown version with rel="alternate" type="text/markdown". It can point to the llms.txt file that covers it with rel="describedby". Both work as HTML <link> elements or as an HTTP Link: header.
How llms.txt differs from robots.txt and sitemaps
These three files do different jobs, and llms.txt replaces neither of the others.
| File | Job | Who reads it |
|---|---|---|
| robots.txt | Says which crawlers may fetch which paths | Search and AI crawlers that honor it |
| sitemap.xml | Lists every page you want indexed | Search engines |
| llms.txt | Points to the few pages that explain the site | AI tools that choose to look for it |
The proposal puts it this way: "While sitemaps list all pages for search engines, llms.txt offers a curated overview for LLMs." It also says the file complements robots.txt by giving context for content a site already allows.
Access control still lives in robots.txt. If you block an AI crawler there, an llms.txt file will not undo it. Our guide to auditing AI crawler access covers those rules. For Google’s plan to honor a shared AI training opt-out in robots.txt, see our post on Google’s shared AI training control.
What Google says about llms.txt
Google’s Search documentation is direct on this point. Its AI features and your website page says: "You don’t need to create new machine readable files, AI text files, or markup to appear in these features." The same page says there are no additional requirements or special optimizations needed to appear in AI Overviews or AI Mode. That page does not name llms.txt, and it was last updated December 10, 2025.
Other parts of Google treat the file as real. Chrome’s Lighthouse tool has an llms.txt audit in its agentic browsing checks. According to Chrome’s documentation, the audit flags a server error when fetching the file. A missing file (a 404) is marked not applicable, because providing one is "optional at the moment." The page says nothing about Google Search ranking.
Google’s Gemini API documentation also publishes its own llms.txt file. It is a long index of documentation pages with one section.
So the Search team says you do not need it, and the developer tools teams use it. Both can be true. Search reads your HTML. Agents and coding tools may read docs on demand, and a short index helps them.
If your goal is controlling how pages show in Google’s AI features, the levers are snippet controls and indexing rules. Our post on noindex vs nosnippet covers which setting does what.
What OpenAI and Anthropic say
Neither company’s crawler documentation says its crawlers read llms.txt. Both companies publish an llms.txt file for their own developer docs.
OpenAI’s overview of its crawlers names OAI-SearchBot (for ChatGPT search), GPTBot (for training), ChatGPT-User (for visits a user’s request triggers) and OAI-AdsBot (for ad landing pages). The page describes robots.txt rules for OAI-SearchBot and GPTBot. It mentions llms.txt only as a pointer to its own documentation index, which has an H1, a summary blockquote and link sections, close to the proposal’s format.
Anthropic’s support article, "Does Anthropic crawl data from the web", names three bots: ClaudeBot, Claude-User and Claude-SearchBot. It says opting out requires changes to robots.txt. The article, updated April 7, 2026, does not mention llms.txt. Anthropic’s developer docs publish an llms.txt file that indexes about 845 English pages.
In short, the AI companies use llms.txt as a doorway into their own documentation. None of the three documents it as a ranking or citation signal for your site.
Who should add an llms.txt file
The file costs little, so the question is less "should we" and more "where does it pay off."
Good fit:
- Documentation sites, API references and developer portals. People query these through AI coding tools, and the AI companies publish the file for their own docs.
- Knowledge bases and help centers with long, stable articles.
- Association sites with standards, guidelines or position papers that members ask AI tools about.
Low priority:
- Small brochure sites with a dozen pages. The home page already does the job.
- News-heavy sites where content changes daily. A hand-built file goes stale fast.
Do first: make sure AI crawlers you want can reach the site, that key pages carry their content in text rather than images or PDFs alone, and that internal links make sense. Our posts on generative engine optimization and answer engine optimization cover that groundwork.
How to add llms.txt in WordPress and other CMSs
Many CMS sites will not need to write the file by hand.
According to llmstxt.org, Yoast SEO, AIOSEO, Wix, Mintlify and GitBook can generate the file. It also lists community plugins, including Drupal LLM Support, docusaurus-plugin-llms and vitepress-plugin-llms.
Yoast’s llms.txt feature page says the feature is free in Yoast SEO, is turned on in the plugin’s site feature settings, and generates and refreshes the file automatically. It also offers a manual mode where you choose the content or write your own.
A few checks apply whatever tool you use:
- Open the file in a browser at
/llms.txtand read it. Auto-generated files can list pages you would not choose. - Run Lighthouse and look for the llms.txt audit. A server error is the failure case Chrome documents.
- Check that linked pages are public. Do not list member-only pages that return a login screen.
- Set a review date. Add the file to your quarterly content review so it does not drift.
If you also want Markdown versions of pages, test that your host serves .md URLs as text. Some caching and security layers block unknown file types.
Limits and open questions
The proposal is a proposal. It is not published by a standards body, and its format has changed once already (v2). Expect more changes.
None of the Google, OpenAI or Anthropic pages we checked gives data on how often their tools read third-party llms.txt files. Claims that the file raises AI citations are unproven. Treat them as directional at best.
The file can also expose structure you would rather not advertise. List only pages you are happy for any tool to read, the same rule you would apply to a public sitemap.
Frequently Asked Questions
What is llms.txt in one sentence?
It is a Markdown file, usually at /llms.txt, that gives AI tools a short summary of a site and links to its most useful pages, proposed by Jeremy Howard in September 2024.
Is llms.txt an official web standard?
No. It is a community proposal published at llmstxt.org, not a standard from a standards body.
Does Google use llms.txt for search rankings or AI Overviews?
Google’s Search documentation says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode. Google has not documented any ranking use of the file.
Do ChatGPT or Claude read my llms.txt file?
OpenAI’s and Anthropic’s crawler documentation does not say their crawlers read it. Both companies control crawling through robots.txt and publish their own llms.txt files for developer docs.
Will llms.txt block AI crawlers?
No. It has no access rules. Use robots.txt to allow or block specific crawlers.
What is the only required part of the file?
An H1 heading with the name of the site or project. Everything else, including the summary blockquote and link lists, is optional.
What does the “Optional” section mean?
It holds secondary links that an AI tool can skip when it needs to keep its context short.
Can a site have more than one llms.txt file?
Yes, under version 2 of the proposal. A file at a subpath, such as /docs/llms.txt, covers pages under that path, and tools should use the most specific file.
Does Yoast SEO create llms.txt for free?
Yes. Yoast’s feature page says the llms.txt generator is free and refreshes the file automatically once you turn it on in site feature settings.
Should a small nonprofit site bother with llms.txt?
Only if it is quick in your CMS. Crawl access, clear page text and good internal links matter more for small sites.