llms.txt is a proposed Markdown file and a form of machine-readable content that gives AI agents and LLM-based tools a concise introduction to a website and a curated route to its most useful resources. It can be understood as a curated, annotated site map for AI systems. Unlike sitemap.xml, which primarily lists URLs for search-engine discovery, llms.txt can explain the site or product and organize selected authoritative resources with short descriptions.
A site usually publishes the main file at /llms.txt, although the current proposal also allows a file within a subpath such as /docs/llms.txt. The file is intentionally selective: it identifies the sources that should be treated as authoritative and explains what each linked page is for instead of reproducing a complete crawl inventory.
Why can llms.txt be useful?
llms.txt gives a website a concise, publisher-controlled map of its content at one predictable URL. The Markdown format is inexpensive to create and easy to update.
That map becomes more useful as a site grows. Important information may be spread across product pages, documentation, API references, policies, tutorials, and support content. Navigation, repeated page furniture, and long pages can also make it harder for an AI crawler or tool to identify the right source quickly.
- Establish the site or product clearly. A short summary can name the entity, explain what it provides, and reduce ambiguity with similarly named products.
- Prioritize authoritative sources. Curated links can direct a tool to current product, documentation, policy, pricing, or reference pages instead of leaving every URL with equal weight.
- Explain how content areas relate. Link descriptions can distinguish a product overview from technical documentation, a specification, an API reference, or an optional background source.
- Reduce discovery work. A compact Markdown index gives a tool a text-first route to useful pages without requiring it to interpret the entire navigation structure first.
How do you create an llms.txt file?
The proposal requires only an H1 naming the site or project. In practice, the most useful files add a short summary and a small number of clearly labeled resource sections.
# Example Company
> Example Company helps teams manage projects and automate routine work.
## Start here
- [Product overview](https://example.com/product): What the product does and who it serves.
- [Documentation](https://example.com/docs): Setup, concepts, and integration guides.
## Reference
- [API reference](https://example.com/docs/api): Endpoints, authentication, and responses.
## Optional
- [Changelog](https://example.com/changelog): Recent product and API changes.- 01 · Identity
Start with one clear H1
Name the site, product, project, or documentation set covered by the file.
- 02 · Summary
Explain the site briefly
Use a blockquote to state what the organization or product does and who the resources serve.
- 03 · Sections
Group links by purpose
Organize resources around real tasks such as getting started, documentation, API reference, or support.
- 04 · Descriptions
Tell agents why each link matters
Use annotated links so a tool can choose the right source without opening every page first.
See how AON structures its llms.txt
AON uses its file to establish the Agent Offer Network entity, explain its role in AI commerce, separate the commercial product from AgentOffer Protocol, and route agents to the website, Glossary, Docs, protocol sources, and policy pages.
# AON - Agent Offer Network for AI Agents
> Agent Offer Network (AON) is a structured offer network for AI commerce.
> It connects AI product builders with merchant offers, matches user intent
> to relevant recommendations, and preserves attribution from click to
> qualified outcome.
## Website page reference
- [AON homepage](https://www.aon.pro/): The primary brand and service overview.
- [AON Glossary](https://www.aon.pro/glossary): Definitions for AON and agentic commerce.
## AON Docs reference
- [AON Docs](https://docs.aon.pro/): Developer and Partner documentation.
- [MCP integration](https://docs.aon.pro/mcp): The hosted MCP integration overview.How is llms.txt different from robots.txt and sitemap.xml?
These files solve different access, discovery, and context problems. A website may publish all three because each one addresses a different reader and uses a different format.
| File | Primary audience | Coverage and role | Format and typical content |
|---|---|---|---|
robots.txt | Conforming web crawlers. | Applies access rules to site paths. It does not inventory or explain the site's content. | A plain-text file containing groups of directives such as User-agent, Allow, Disallow, and optional Sitemap locations. |
sitemap.xml | Search engines and their crawlers. | Provides a broad inventory of canonical or indexable URLs that a publisher wants search engines to discover. | An XML file or sitemap index focused on page URLs, with optional metadata such as the last meaningful modification date. |
llms.txt | AI agents and LLM-based tools. | Introduces the site or product and curates high-signal authoritative resources instead of listing every eligible URL. | A Markdown file with an H1, an optional summary and explanatory context, plus grouped, annotated links. |
llms.txt complements established crawl and discovery files. It does not replace crawler access rules, submit URLs for indexing, or guarantee that an AI system will use or cite a page.
Who should publish an llms.txt file?
Documentation sites, APIs, developer platforms, SaaS products, technical knowledge bases, and companies with several distinct content areas have the clearest use case. These sites benefit most from stating which sources are authoritative and what each source is designed to answer.
Publishing llms.txt is a reasonable low-cost addition for these sites. It requires no JavaScript, API integration, or new crawler permission model. A team can start with one reviewed Markdown file and expand it only when the information architecture requires more detail.
The following established technology companies publish llms.txt files to help AI tools navigate and interpret their documentation and product resources. These examples show how the format can serve different content sets and audiences:
OpenAI Developers' llms.txt: Routes agents across OpenAI API, Codex, commerce, Cookbook, and other developer documentation sets.
Anthropic Documentation's llms.txt: Publishes an index of Claude developer documentation and points to a larger full-document export.
Firecrawl's llms.txt: Maps its web rendering, cleaning, full-page retrieval, crawling, parsing, and structured-extraction tools to detailed documentation.
Google Gemini API Docs' llms.txt: Provides a Markdown index for Gemini API documentation and machine-readable page versions.
Cloudflare Developers' llms.txt: Uses a top-level developer index that points to product-specific llms.txt files across its platform.
Vercel's llms.txt: Organizes product, deployment, AI, API, and agent resources around common developer tasks.
Stripe Documentation's llms.txt: Maps a large payments and financial-infrastructure documentation set into annotated Markdown links.
A small website with only a few well-linked pages may gain less from a detailed file. In that case, clear navigation, focused page copy, and accurate metadata should remain the priority.
What are the best practices for llms.txt?
- Write a concise and factual description of the site instead of a marketing slogan.
- Link canonical, public, and current pages that answer distinct user questions.
- Put the most authoritative and frequently needed resources first.
- Describe what each link contains and when an agent should use it.
- Separate secondary resources under an
Optionalsection when useful. - Review the file after major launches, documentation changes, migrations, or URL updates.
- Serve it successfully as readable Markdown and verify every linked resource.
Questions about llms.txt
Is llms.txt an official web standard?
No. llms.txt is a proposed community format rather than a formal W3C or IETF standard. Its simple Markdown structure has nevertheless been adopted by a growing number of developer platforms and documentation sites.
Is an llms.txt file expensive to create or maintain?
Usually not. A basic implementation is one static Markdown file containing a short description and curated links. The main ongoing cost is reviewing it when important pages, product names, or URLs change.
What should a useful llms.txt file contain?
Start with an H1 naming the site, add a concise summary, then group a limited set of authoritative links under clear H2 headings. Give each link a description that tells an agent why and when to use that source.
Does llms.txt replace robots.txt or sitemap.xml?
No. robots.txt communicates crawler access rules. sitemap.xml supports search-engine discovery with a broad XML inventory of canonical or indexable URLs. llms.txt serves a different purpose: it gives AI agents selected site or product context and annotated routes to authoritative resources.
Does llms.txt improve a website's SEO or GEO?
Not directly. llms.txt is not a confirmed ranking factor for search engines or answer engines, and publishing one does not by itself improve rankings, AI visibility, or citations. It can support generative engine optimization (GEO) on documentation-heavy sites when relevant AI systems retrieve and use it as a guide to authoritative pages. Prioritize accessible, accurate, well-structured pages first; add a concise llms.txt afterward and measure its actual use.
Do AI systems have to read llms.txt?
No. llms.txt does not require ChatGPT, Google AI Overviews, or any other AI system to retrieve the file. It is a concise source that a system may use when it chooses to inspect a site's resources.
Sources
- llms.txt proposal and formatPrimary proposal
- Answer.AI: The /llms.txt fileOriginal publication
- Firecrawl: How to create an llms.txt fileIndustry guide
- Mintlify llms.txt documentationOfficial documentation
- Chrome Lighthouse llms.txt auditOfficial documentation