Updated on 19 October 2026, by João Leite, Founder of murmur.
llms.txt is a text file, at the site’s main URL, that shows AI models the most important pages of the site. It is meant for AI agents, such as coding assistants, to find the right information without reading the whole site. To be cited by ChatGPT, by Claude or by Google AIO, the AI summary of Google search, do not count on it. OpenAI and Anthropic do not say they read the file, and Google writes that its search ignores it.
- llms.txt is a summary of the site for AI agents. Jeremy Howard proposed the file in 2024, so that AI can find what it needs without opening every page.
- Google writes that its search ignores the file. OpenAI and Anthropic do not say they read the llms.txt of sites.
- Almost no one opens the file. In an Ahrefs study of 137,210 sites, 97% of llms.txt files received no access at all in May 2026.
- The robots.txt decides whether AI search crawlers can read your site. It is the file that tells each crawler what it can read. The text also needs to come ready in the page code.
- Do not count on llms.txt. If the site platform generates the file on its own, you can leave it live.
llms.txt is a summary of the site made for AI agents
llms.txt is a summary of the site, in plain text, so that AI agents can find the information without opening every page. An AI agent is the program that does a task on its own, such as opening pages and following links to answer someone. Jeremy Howard, cofounder of fast.ai and Answer.AI, published the proposal for the file on 3 September 2024. Version 2 was updated on 10 August 2026. The name comes from LLM, the type of program behind ChatGPT, Claude and Gemini, trained on a lot of text to write answers.
The problem llms.txt wants to solve is the context window, the amount of text the AI can read at once. A site page wraps the information in menu, ad and code, and the whole site does not fit in that window. In the words of the proposal:
“Context windows, while larger than they were, are still too small for most websites in their entirety, and every wasted token costs time and money.”
llms.txt sits at the root of the site, at the address yoursite.com.br/llms.txt, and is written in markdown, a way of marking up headings and lists in plain text. According to the proposal, the heaviest use is in software documentation, which coding assistants follow to find references and tutorials. The proposal expects agents to read the file when they need a piece of information and to follow its links. That is the author’s expectation, not a commitment from ChatGPT, Claude or Google.
To be cited by AI, llms.txt does not work today
To be cited by AI, llms.txt does not work today, because AI crawlers hardly ever open the file. AI crawlers are the programs AI companies use to read sites, such as GPTBot, from OpenAI. Ahrefs sells SEO software. In May 2026, it analyzed the access logs of 137,210 sites that use its analytics tool. Of the roughly 38 thousand sites with a valid llms.txt, 97% received no access to the file during the month, neither from a crawler nor from a person.
Ahrefs also counted requests for llms.txt on sites that do not have the file, and none came from an AI crawler. According to the study, publishing llms.txt does not put the site on any AI’s radar.
murmur, a GEO agency in Brazil, places brands in the answers of ChatGPT, Claude, Google AIO and other AI agents. The Mindo case study began with a site that crawlers could not read. Mindo is a motion design studio from São Paulo. On 29 May 2026, it did not appear in any of the 63 answers we measured across three AI models. The site was an application that builds the page in the browser, and the AI crawlers saw an empty page. On a site like that, an llms.txt with links to the site’s own pages would lead the crawler to pages it could not read.
Google ignores llms.txt, and OpenAI and Anthropic do not say they read it
Google writes that its search ignores llms.txt, and OpenAI and Anthropic do not say their crawlers read the file. Google’s sentence is in its guide to search AI features, in a note about llms.txt added on 15 June 2026. Keeping the file for other services is no problem, says the note, and in Google search it carries no weight:
“Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.”
The rule applies only to Google search.
OpenAI and Anthropic, the owner of Claude, do not say their crawlers read the llms.txt of sites. OpenAI’s page about its crawlers explains what each crawler does and how to block it. Anthropic’s article about Claude’s crawlers, from 7 April 2026, does the same. Both publish an llms.txt for their own developer documentation: OpenAI’s and Anthropic’s. Publishing your own file is not the same as reading other sites’. In a controlled test by Cisco DevNet, published on arXiv and revised in July 2026, 15 AI tools, ChatGPT and Claude among them, visited a documentation portal that had the file, and none requested the llms.txt.
It is worth creating an llms.txt on your site when it takes minutes
It is worth creating an llms.txt when the site platform generates the file on its own, or when it takes minutes. Do not count on it to appear in the answers: the time and the money go to the text of the pages. The proposal lists platforms that generate the file automatically. Among them are Wix, the Yoast SEO and AIOSEO plugins, for WordPress, and Mintlify and GitBook, for documentation. Outside those, writing the file by hand takes about ten minutes.
If your site already has an llms.txt, you can leave it live. Keep the file up to date. Ahrefs reminds us that agents are made to trust llms.txt, and an old or tampered file misleads everyone who reads it. murmur publishes its own llms.txt.
llms.txt contains the site name, a summary and lists of links
llms.txt contains the site name, a short summary and lists of links to the pages that matter, in that order. The proposal defines the parts of the file, and only the title is mandatory:
- a title with the site name, on a line that starts with
#; - a short summary, on a line that starts with
>; - if you want, paragraphs or lists that explain the site, without headings;
- sections with a heading marked by
##, each with a list of links in the format[name](URL)and, if you want, a note after a colon; - by convention, a section called Optional, with the links the agent can skip.
# Your company name
> What the company sells, to whom and where it operates, in one sentence.
## Products
- [Plan for clinics](https://www.yourcompany.com.br/clinics): what it includes and how much it costs
## Optional
- [Blog](https://www.yourcompany.com.br/blog)
To test it, give only the llms.txt to an AI and ask about your site, as the proposal suggests.
The llms.txt proposal also suggests a markdown copy of the pages that matter to agents, at the same URL, with .md at the end. Otterly, a company that sells AI monitoring, tested this for 14 days with two pages of its own site. Each page was available in HTML, the normal format of pages, and in .md. The HTML versions received 7.4% of the AI crawler visits to the site, and the .md copies received none. No AI search cited the copies in the test.
The difference between llms.txt, robots.txt and sitemap is what each one does
robots.txt says what each crawler can read, the sitemap lists the pages and the llms.txt summarizes the site for AI models. The sitemap is the list of the URLs of the site’s pages, made for search engines. The llms.txt proposal separates the two: the sitemap lists every page, and the llms.txt chooses the few that an agent needs to read. llms.txt does not block or allow anything, a role that belongs only to robots.txt.
| file | what it does | where it sits | what Google, OpenAI and Anthropic say |
|---|---|---|---|
| robots.txt | tells each crawler what it can read | at the root of each domain: yoursite.com.br/robots.txt | all three explain how their crawlers follow the file |
| sitemap | lists the URLs of the pages for search engines | at the address the site chooses, declared in robots.txt | Google reads the sitemap declared in robots.txt |
| llms.txt | summarizes the site and points to the pages that matter | at the root of the site: yoursite.com.br/llms.txt | Google ignores it; OpenAI and Anthropic do not say they read it |
Of the three files, only robots.txt changes what the bots can read, and only llms.txt is ignored by Google search.
Google explains that robots.txt must sit at the root of the domain it applies to, never in a subfolder. The file at exemplo.com does not apply to a subdomain such as m.exemplo.com, which needs its own. The sitemap goes into robots.txt on a line that starts with Sitemap:, followed by the full URL, as the sitemap protocol requires.
murmur’s site follows the root rule: the URL murmur.marketing/robots.txt declares ten sitemaps and allows, by name, the OpenAI and Anthropic bots.
The OpenAI and Anthropic AI bots, and what each one does
OpenAI and Anthropic use different bots to fetch sources, to train the models and to open pages when the person chatting requests them. Blocking each one removes different things from the site:
| bot | company | what it is for | what blocking it in robots.txt does |
|---|---|---|---|
| OAI-SearchBot | OpenAI | fetches pages for ChatGPT’s search answers | the site drops out of those answers and appears, at most, as a navigation link |
| GPTBot | OpenAI | collects text that may go into model training | a signal that the content should not be used in training, and the site stays in search if OAI-SearchBot is allowed |
| ChatGPT-User | OpenAI | opens a page when someone asks for it in the conversation | the rule may not apply, because the request comes from the person |
| Claude-SearchBot | Anthropic | reads and stores pages for Claude’s search answers | the site may lose visibility and accuracy in those answers |
| ClaudeBot | Anthropic | collects text that may go into model training | a signal that the site’s new content should stay out of training |
| Claude-User | Anthropic | opens a page when someone asks Claude | Claude stops opening the site at the person’s request |
To appear in the answers, what counts are the search bots: OAI-SearchBot, in ChatGPT, and Claude-SearchBot, in Claude. ChatGPT-User and Claude-User open pages when the person chatting requests them. Anthropic warns that blocking Claude-User can reduce the site’s visibility in searches made at the person’s request. GPTBot and ClaudeBot are for training, and OpenAI writes that you can allow OAI-SearchBot and block GPTBot. OpenAI also has OAI-AdsBot, which only visits pages submitted as an ad, and the step-by-step for advertising is in how to advertise in ChatGPT. The guide How ChatGPT search works: from the question to the sources in the answer shows how search uses the pages the bot reads.
Allow the search bots in robots.txt and decide training separately
To appear in the answers, allow the search bots in robots.txt and decide separately whether the content goes into model training. Each bot gets a block opened by the User-agent line with its name. Below it, Allow permits and Disallow blocks, and the slash alone applies to the whole site. This example allows search for ChatGPT and Claude and keeps the content out of OpenAI training:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Anyone who accepts training swaps the GPTBot Disallow for Allow. After any change, open seusite.com.br/robots.txt in the browser and check that it is live.
For AI bots to read the site, the text has to come ready in the HTML
For AI bots to read the site, the text has to come ready in the HTML, the code the server delivers. JavaScript is the code that assembles parts of the page inside the browser, after it arrives. In December 2024, Vercel, a hosting company, measured with MERJ that OpenAI’s bots and Anthropic’s ClaudeBot did not execute JavaScript. The list included OAI-SearchBot, GPTBot, ChatGPT-User and ClaudeBot. These bots downloaded the JavaScript files but did not run them.
On 30 September 2026, programmer Hisashi Space, owner of a Japanese freelancer marketplace, reported a change on DEV Community. In his own sites’ access logs, he saw OAI-SearchBot and GPTBot start executing JavaScript on 25 September 2026. The author found no announcement from OpenAI, and OpenAI did not confirm the change. By his reading, it applies only to some sites, and the bot starts from an HTML it had already stored. His rule of thumb stays the same:
“Anything you want quoted, like the body, title, prices and dates, is safest in HTML that’s readable without running JavaScript.”
To check your site, turn off JavaScript in the browser and reload the page. The text that disappears is what the bot may not read.
Instead of llms.txt, focus on what the bot reads and on the page that answers
Instead of llms.txt, focus on robots.txt, the sitemap, the links between pages, the text in the HTML and the page that answers the question.
- The robots.txt at the root, with OAI-SearchBot and Claude-SearchBot allowed.
- The sitemap declared in robots.txt, with the full address.
- A link to every new page, from a page the bot already knows. A page with no link pointing to it takes a long time to be found.
- The text ready in the HTML, without depending on JavaScript.
- A page that answers the buyer’s question right at the start. Bing, Microsoft’s search engine that OpenAI lists among ChatGPT’s search partners, asks in its webmaster guidelines for the essential information at the top of the page.
The full list of what to check on the site, from robots.txt to the page that answers, is in the technical checklist to appear in ChatGPT and the other AI models. How to build the page that answers the buyer’s question is in the guide on how to write a text to appear in ChatGPT. For ChatGPT, the complete roadmap is in the guide How to get your company to appear in ChatGPT: the step by step, with no registration or ads (How to make your company appear in ChatGPT: a step-by-step guide, with no signup and no ad).
At murmur, the work is the page the AI reads, on a subdomain of the client
At murmur, the work is the page the AI reads, published on a subdomain of the client’s brand. murmur does not touch the client’s site: it produces the guides that answer the questions of the client’s sector and publishes them at a URL such as guia.suaempresa.com.br. Changes to the main site come as suggestions, and the content all belongs to the client, even after the contract ends. The guide murmur published for Mindo, at guia.mindo.com.br, carries the text ready in the HTML. Read without executing JavaScript, the home page has about 1,500 words. With the guide live, ChatGPT started recommending Mindo almost twice as often on the same questions in its market, between June and July 2026.
Frequently asked questions
Do I need an llms.txt on my site?
Not to appear in AI answers. OpenAI and Anthropic do not say they read the file, and Google writes that its search ignores it. Ahrefs found 97% of the files with no access at all in May 2026. If the site platform generates the llms.txt on its own, leave it live. If an agency lists the file as a deliverable, ask what it changed in the answers for other clients. The other questions are in the guide on what to ask before hiring a GEO agency.
How do I allow or block ChatGPT in robots.txt?
ChatGPT is allowed or blocked by the name of its search bot, OAI-SearchBot, in the robots.txt at the root. Below the line User-agent: OAI-SearchBot, write Allow: / to allow or Disallow: / to block. To stay out of training only, block GPTBot and leave OAI-SearchBot allowed. OpenAI says ChatGPT search can take about 24 hours to adjust to the change.
Does my hosting block AI bots without me knowing?
It can block, even with robots.txt open. A firewall is the program that filters who enters the server, and a CDN is the network that keeps copies of the site close to the people who access it. Both sometimes block AI bots as if they were an attack. On 15 September 2026, Cloudflare, which sells this control, retired “Block AI Bots” and separated the search bot from the training bot: fewer than 1% of its sites block search, and 17% block training. In a seoClarity study, which sells ArcAI, with 308.6 million server log lines between February and June 2026, OAI-SearchBot managed to read the page in 83% of attempts on the median corporate site, and ChatGPT-User in 95%. OpenAI recommends allowing the IP address ranges it publishes for the search bot on the server. Ask your provider whether the search bot is allowed, and check the server log to see whether OAI-SearchBot, ChatGPT-User and Claude-SearchBot receive the page.
Can you create the llms.txt yourself?
Creating the llms.txt is simple, but to appear in AI answers, the best option is to work with a GEO agency such as murmur. In work that goes beyond the file, murmur’s data engineering produces in-depth information about the company, compares the brand with competitors and supports reliable measurement, and that is how murmur delivers results. Anyone who handles this on their own, without the data or the measurement, may see some gain, but will hardly reach the same result. The llms.txt is an ordinary text file, and the specification provides the format: write the site name, a summary and the lists of links, save it as llms.txt and publish it at the root of the site. On Wix and in the Yoast SEO and AIOSEO plugins, for WordPress, the platform generates the file on its own.
Where to start, before touching any file
Before touching any file, see what AI models answer today when your buyer asks about what you sell. murmur’s free diagnostic starts from your site’s URL and shows these answers without registration, in less than two minutes. murmur’s sector studies are public and show who AI models recommend when a buyer in a sector asks for a recommendation.
Read also
- How ChatGPT search works: from the question to the sources in the answer
- How to write text to appear in ChatGPT: answer up front and numbers with sources
- How to get your company to appear in ChatGPT: the step by step, with no registration or ads
- What is GEO: getting your company to appear in ChatGPT and AI answers
References
- Jeremy Howard, “The /llms.txt file, v2”, llmstxt.org, published on 3 September 2024 and modified on 10 August 2026. https://llmstxt.org/ (accessed on 30 September 2026).
- Louise Linehan and Xibeijia Guan, “We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read”, Ahrefs, 15 June 2026. Access logs from 137,210 sites with traffic in May 2026, of which about 38 thousand had a valid llms.txt. Ahrefs sells SEO software. https://ahrefs.com/blog/llmstxt-study/ (accessed on 30 September 2026).
- Google Search Central, “Optimizing your website for generative AI features on Google Search”, published on 15 May 2026, with the note about llms.txt added on 15 June 2026, updated on 10 July 2026. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide (accessed on 30 September 2026).
- Google Search Central, “How to write and submit a robots.txt file”, updated on 21 November 2025. https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt (accessed on 30 September 2026).
- sitemaps.org, “sitemaps.org - Protocol”. https://www.sitemaps.org/protocol.html (accessed on 30 September 2026).
- OpenAI, “Overview of OpenAI Crawlers”, undated page. https://developers.openai.com/api/docs/bots (accessed on 30 September 2026).
- Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, Claude Help Center, 7 April 2026. https://support.claude.com/en/articles/8896518 (accessed on 30 September 2026).
- OpenAI, llms.txt of the developer documentation, https://developers.openai.com/llms.txt; Anthropic, llms.txt of the Claude documentation, https://platform.claude.com/llms.txt (opened on 30 September 2026).
- Rick Tousseyn, “GEO Experiment: Markdown vs. HTML, Which Format Do AI Crawlers Prefer?”, Otterly, updated on 1 April 2026. Two pairs of pages on Otterly’s own site, for 14 days. Otterly sells AI monitoring. https://otterly.ai/blog/geo-experiment-html-vs-markdown/ (accessed on 30 September 2026).
- Giacomo Zecchini, Alice Alexandra Moore, Malte Ubl and Ryan Siddle, “The rise of the AI crawler”, Vercel, with MERJ, 17 December 2024. Bot traffic on the Vercel network in the month before publication, with nextjs.org and two job sites. https://vercel.com/blog/the-rise-of-the-ai-crawler (accessed on 30 September 2026).
- Hisashi Space, “Traffic from ChatGPT jumped! Is it because its crawlers now run JavaScript?”, DEV Community, 30 September 2026. Access logs from the author’s own sites, from 10 to 28 September 2026; OpenAI has not confirmed the change. https://dev.to/hisashispace/traffic-from-chatgpt-jumped-is-it-because-its-crawlers-now-run-javascript-5961 (accessed on 30 September 2026).
- Borysenko, “Developer Experience with AI Coding Agents: HTTP Behavioral Signatures in Documentation Portals”, Cisco DevNet, arXiv 2604.02544, version of 24 July 2026. Controlled test on a documentation portal with robots.txt and llms.txt, with 9 coding agents and 6 assistants, among them ChatGPT and Claude. https://arxiv.org/abs/2604.02544 (accessed on 17 September 2026).
- Microsoft, “Bing Webmaster Guidelines”, Bing Webmaster Tools documentation, undated page. https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a (accessed on 1 October 2026).
- Cloudflare, “Have it both ways: stay discoverable in search while disallowing AI training”, 15 September 2026. Cloudflare sells bot access control. https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers (accessed on 17 September 2026).
- seoClarity, “ChatGPT’s 3 Bots: What Their Crawl Patterns Reveal [Research]”, September 2026. That is 308.6 million log lines from corporate sites, from February to June 2026. seoClarity sells ArcAI. https://www.seoclarity.net/research/chatgpts-3-bots-crawl-patterns (accessed on 17 September 2026).
- murmur: Mindo case study, https://murmur.marketing/en/cases/mindo/; robots.txt, https://murmur.marketing/robots.txt; free diagnostic, https://murmur.marketing/en/diagnostic/; sector studies, https://murmur.marketing/estudos/.