What is llms.txt?

llms.txt is a Markdown file at the root of a website that tells a language model or an AI agent what the site is and where its content is. Its address is /llms.txt, next to robots.txt and the sitemap. Jeremy Howard published the format as a proposal in September 2024. The documentation is at llmstxt.org. No standards body has adopted it.

What is in an llms.txt file

The proposal sets the order of the parts:

• a heading with the name of the site or project, the one required part
• a quoted paragraph with a short summary
• free text with notes on how to read the site
• sections with lists of links, one page per line, with an optional note after a colon

A section with the heading “Optional” holds links that an agent can skip when it has little room. The proposal also suggests a Markdown copy of a page at the same address with .md appended. The file is written in Markdown because language models read that format well. The fixed order lets a program parse it too.

What llms.txt is used for

An agent that helps a user fetches the file first. It finds the page that holds the answer and opens only that page. A web page wraps its text in navigation and scripts, and a language model has limited room for text. A short guide saves the agent from reading a whole site.

According to llmstxt.org, the file is used most for software documentation. Coding agents follow it there to references and tutorials.

llms.txt, robots.txt and the sitemap

llms.txt grants no access and blocks none. It cannot let in a crawler that robots.txt locks out.

Does llms.txt help in search?

Google states that its AI features need no AI text files and no special markup (Google, updated December 10, 2025). According to llmstxt.org, OpenAI and Anthropic publish llms.txt files for their own developer documentation. That says nothing about whether an assistant reads the file of a company website before it answers. Nobody can promise a ranking or a citation from the file.

Both GEO and SEO start with a crawler that can fetch a page and read its text in the HTML. An llms.txt is a guide on top of that. It repairs no page that a crawler cannot read.

The llms.txt of this site

I am an IT expert for data platforms in Frankfurt and have built websites for more than twenty years. This site is my test case, and its llms.txt is part of it. A script builds the file from the content of the site with each build. A new page appears in the file without a second step.

The file opens with my name, a summary with role and city, and the address of the contact page. A section tells an agent where to look: the work pages for cases, the background pages for career stages. Then comes one line per page with title, address and description. Blog posts carry their publishing date, newest first. The German pages stand at the end. The file asks an agent to quote a number with its date and to link the page a fact comes from.

Two points differ from the proposal. The links lead to the HTML pages, because the site has no Markdown copies. The file also lists the pages in full, where the proposal describes a curated overview. The robots.txt of the site names the file in its first line.

Claude and Codex write much of the code of this site under rules that a machine checks, and I review the result. My audit of a company’s AI search visibility records what the assistants say about the company. It also shows whether the company’s site should get an llms.txt. I take on this work as a freelancer or as a permanent employee.