← back to the archiveCover illustration for “llms.txt: a site map written for language models”
TERM6mo ago·Published · Updated ·by Andy Padia

llms.txt: a site map written for language models

In short: llms.txt is a proposed Markdown file at a site's root that gives AI tools a short summary and links to key pages, with no control over crawling.

llms.txt is a proposed convention, published by Jeremy Howard on 3 September 2024, for a Markdown file at /llms.txt that gives language models and AI tools a concise summary of a site and links to its most useful pages, ideally clean Markdown versions. The proposal expects the file to be read at inference time, when a tool is answering a question. It does not control crawling or training.

llms.txt in plain words

A short "start here" note left at the front door for AI visitors: what this place is and which rooms are worth visiting.

How it works

The only required element is a title naming the site or project. After it come an optional short summary in a blockquote, optional detail paragraphs, and sections of links with brief notes. A section named Optional is reserved, by convention, for secondary information. The proposal also recommends publishing a clean .md version of each page.

An AI tool, coding agent or assistant that knows the convention can fetch the file to understand the site quickly. Nothing forces any tool to read it.

Example

As an illustration, a minimal file:

# AndyMental
> Daily notes on shipping AI in production: agents, evals, governance.

## Field guide
- [Agent harness](https://andymental.com/drops/agent-harness): the code that runs an agent loop
- [Evals](https://andymental.com/drops/evals): testing AI systems before release

Where you'll meet it

  • Developer documentation sites that want coding agents to read them well
  • Personal and company sites experimenting with AI discoverability
  • Coding agents and assistants that fetch documentation on demand
  • Discussions about GEO and AI search

What it changes for you

For documentation teams, it is a cheap way to make docs easier for coding agents to consume. For marketing teams, it is not a ranking lever: Google's guidance on AI features, read in November 2026, says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode. Treat it as a courtesy file, maintained with your sitemap.

llms.txt vs robots.txt

robots.txt tells crawlers what they may access. llms.txt suggests what is worth reading. One is permission, the other is a guidebook.

Common mistake

My view: expecting llms.txt to produce AI citations. Citations come from clear, well-sourced pages that answer questions. The file helps tools find those pages; it cannot make thin pages worth citing.

Generative engine optimization · Answer engine optimization · AI citations · AI Overviews

From the archive: A package named in official docs still needs an identity check

llms.txt helps tools find your best pages, and only clear, well-sourced pages earn the citation.

#field-guide#ai-search#seo#geo#web-standards

Sources

  1. llmstxt.org
  2. developers.google.com/search/docs/appearance/ai-features
← older drop
Batch inference: cheaper AI for work that can wait
newer drop →
A2A protocol: how AI agents talk to other agents

related drops

explore all 557 drops →
← back to the archiveday 175