Updated on 19 October 2026, by João Leite, Founder of murmur.
The technical checklist to appear in ChatGPT checks whether the AI bot can enter the site, read the text and find each page. The list also applies to Claude, Gemini, Perplexity and Google AIO, the AI-written summary that appears at the top of many Google searches. There are 23 items in five blocks: bot access, readable text, page found, page that answers and company recognized. The llms.txt stays out of the list, because AI bots almost never open the file, and the schema enters only to describe what is on the screen.
- Without the search bot allowed, the site is no longer shown in the answers. According to OpenAI, a site closed to OAI-SearchBot does not appear in ChatGPT answers with search and only appears as a navigation link.
- The text has to arrive ready in the HTML. OpenAI’s bots and Anthropic’s ClaudeBot were not executing JavaScript in Vercel’s December 2024 measurement.
- In Google AIO, only a page that Google has already stored in the index enters. Google asks for the page to be indexed and eligible to appear in search with a snippet of text.
- The robots.txt does not apply equally to all AI models. In a July 2026 study, with 200 attempts across ten assistants, Claude respected the block and Gemini opened the blocked pages.
- The llms.txt is almost never opened, and the schema only describes what is on the screen. Ahrefs found 97% of valid llms.txt files with no access at all in May 2026, and none of the five AI models tested by searchVIU found the price placed only in the markup.
murmur, a GEO agency in Brazil that places brands in the answers of ChatGPT, Claude, Google AIO and other AI agents, publishes this checklist. With each client, murmur measures where the brand is missing in the answers, produces and structures the pages that answer the buyer and tracks the change week by week.
Bot access: the robots.txt and the hosting let the AI in
Access comes first in the checklist, because the AI only cites the page its search bot can open. The AI bot (crawler) is the automatic visitor that each AI company sends to sites to read the pages. The robots.txt is the file, at the site root, with the rule for each bot.
- The robots.txt sits at the root of each address. Google writes that the file applies only to the address at whose root it sits, and that is why blog.yourcompany.com needs its own file. Open yoursite.com/robots.txt in the browser and check that it responds.
- OAI-SearchBot, ChatGPT’s search crawler, is allowed. OpenAI writes that a site closed to it does not appear in ChatGPT answers with search, and a change to the file can take about 24 hours to take effect in ChatGPT search. Model training is decided separately, as the guide on OpenAI’s and Anthropic’s AI crawlers explains.
- Claude-SearchBot, Claude’s search crawler, is allowed. According to Anthropic, anyone who turns off this crawler can lose visibility and accuracy in Claude’s search results.
- PerplexityBot is allowed, and the server accepts its IP ranges. Perplexity’s documentation recommends both things for the site to appear in its search, and the step-by-step instructions are in the guide on how to appear in Perplexity.
- Googlebot is allowed, and Google-Extended is a separate decision. For Google, AI is part of Search, and the robots.txt rules for Googlebot control its access to the site. Google-Extended controls something else: the training of the next Gemini models and the use of the text in the answers of the Gemini app.
- The hosting, the firewall and the CDN let the AI bots through. These protection services sometimes treat the bot as an attack, even with the robots.txt open. On 15 September 2026, Cloudflare, which sells this control, retired “Block AI Bots” and separated the search bot from the training bot: fewer than 1% of its sites block search, and 17% block training. Even without a block, not every visit gets through: in a seoClarity study, which sells SEO and AI search monitoring tools, with 308.6 million log lines from corporate sites between February and June 2026, OAI-SearchBot managed to read the page in 83% of attempts on the median site, and ChatGPT-User in 95%. Ask whoever handles the hosting whether the search bot is allowed, and check in the server log whether OAI-SearchBot and ChatGPT-User receive the page. How to find this block is covered in the guide on hosting that blocks AI bots.
The robots.txt does not hold back all AI models the same way. In a study published in July 2026, Gabriel Lopez-Fonseca and colleagues, from the Polytechnic University of Madrid and ETH Zurich, tested ten AI assistants. In the 200 attempts, 20 per assistant, Claude opened the allowed pages and none of the blocked ones, and Gemini, DeepSeek, Grok and Qwen opened allowed and blocked pages. The test measured the assistant opening the page at the request of the person chatting, not the crawler that builds the search index. The authors’ conclusion:
“these findings show that robots.txt cannot be assumed to reliably govern inference-time web access by AI assistants”
For anyone who wants to appear, the reading is the reverse: in Claude, a forgotten block in the robots.txt bars the page, even when someone asks the assistant to open it.
Readable text: what the robot reads without executing JavaScript
The text the AI can cite is what arrives ready in the HTML, the code the server delivers before any script runs. JavaScript, in this case, is the code that finishes building the page in the browser of the person opening it.
- The answer, the price and the date are in the HTML the server delivers. In December 2024, Vercel and MERJ saw OpenAI’s crawlers and Anthropic’s ClaudeBot download the JavaScript without executing it. That measurement has not been repeated since, so today’s test is done by hand: with the page open, Ctrl+U shows the source code, and the answer, the date and the price must appear in it. The step-by-step instructions are in the guide on text ready in the HTML.
- Nothing that answers is behind a tab, accordion, button or image. If the text only loads after the click, the crawler that does not execute JavaScript does not receive it. In an Otterly test with five AI models, a number placed only inside an image was not read by any of them.
murmur found this defect in the diagnostic for Mindo, a motion design studio in São Paulo. On 29 May 2026, Mindo was absent from all 63 answers we measured across three AI models, and its site, built in the browser, reached the robot blank. Allowing any robot would not have solved that, because there was no text to read.
Page found: Google index, sitemap and links between pages
The AI only cites the page it finds, and in Google AIO the page enters through the Google index.
- The page appears in regular Google search, with a title and a snippet. For a page to become a supporting link in AI Overview and AI Mode, Google requires the page to be indexed and eligible to show a snippet. The URL inspection tool in Search Console tells you whether the page is in the index.
- No noindex or nosnippet left behind on pages that should appear. These are the tags that ask Google not to list the page or not to show a snippet from it. Google lists noindex, nosnippet, data-nosnippet and max-snippet as the controls that limit what the page shows in Search. Since AI is part of Search, according to Google, the rule also applies to AI Overview.
- The site is included in the generative AI features in Search Console. Since 31 August 2026, Search Console has a control for generative AI in Search, and including the site is the default. Check in Settings whether someone changed it. What else counts in Google AIO is in the guide about AI Overview and AI Mode.
- The sitemap contains every page that should be found, and robots.txt points to it. The sitemap is the file that hands search engines all the site’s URLs at once. The sitemaps protocol allows you to point to the file in robots.txt, on a line that starts with Sitemap: and carries the full URL.
- The date of each page in the sitemap is the date of the last real change. The lastmod field holds that date, and Google only uses it when it matches the page.
- Each new page gets a link from a page the search engine already knows. Google explains that it discovers new pages through links from pages it has already visited and through the sitemap. A page with no links at all depends on the sitemap or on links from other sites.
- Each page has a single URL. When the same page opens at two URLs, the redirect and the canonical tag, which points to the main URL, tell Google which one to keep. Google recommends these methods and writes that none of them is mandatory.
- The site is verified in Search Console and Bing Webmaster Tools, with the sitemap submitted. Both dashboards show which pages Google and Bing have kept. What each one shows about AI models is in the guide about Search Console and Bing Webmaster Tools.
- The site notifies Bing through IndexNow when a page is added, changed or removed. IndexNow is the notice the site sends to participating search engines instead of waiting for them to come by. According to the official IndexNow list, the notice reaches Bing, Yandex, Naver, Seznam.cz, Yep and Amazon. Google is not on it. OpenAI names Microsoft among ChatGPT’s search providers and does not say which questions each one handles, the topic of the guide about ChatGPT search.
About the sitemap date, Google’s page on how to create the file says:
O Google usa o valor <lastmod> se ele estiver correto de maneira consistente e comprovável (por exemplo, com base na comparação com a última data de modificação da página).
Original quote in Portuguese, preserved as published. In English: “Google uses the `<lastmod>` value if it is consistently and verifiably correct (for example, based on a comparison with the page's last modification date).”
On the same page, changing only the year in the copyright notice does not count as an update, while changing the main text, the structured data or the links does.
A page that answers: the answer up front and one topic per page
A page that answers puts the answer up front, in the words of the person asking, with the fact written in the text itself. Bing belongs to Microsoft, which OpenAI lists among ChatGPT’s search providers, and Bing’s guidelines for webmasters say a page is more likely to become a source when it can be understood without support from other pages, has the essentials right at the top and does not mix unrelated topics.
- The first sentence answers the question, and the title and headings use the buyer’s words. The step-by-step is in the guide about how to write a text to appear in ChatGPT. What makes a passage easy to cite is in the guide about AI citability.
- Price, delivery time, number and date are written in the text, not only in the image or the markup. None of the five AI models tested by searchVIU in October 2025 found the price placed only in the page’s markup.
- The update date appears on screen and is the same as in the markup. Google’s rule for dates has two parts: a visible label, such as “Updated on”, and the same value in the structured data.
- The schema, if there is one, describes only what is on screen. Schema is a block in the page code that describes the name, price or address for machines.
A recognized company: the same name everywhere and the company’s pages
AI models confuse companies with similar names, and the defense is to use the same name, together with what the company does, everywhere.
- The same name and description on the site, on LinkedIn and on the company’s profiles. The name carries what sets the company apart from another with the same name. Profile by profile, the step-by-step guide is in the guide about the same name and details everywhere.
- The site has the pages the buyer looks for: the service, the price or what it depends on, the case studies and who is responsible for the company. The calculation is one page for each genuinely different buyer question, with the complete answer, not one for each way of writing the same question. How many pages to make, and in what order, appears in the guide about how many pages to publish to appear in ChatGPT.
What each item is worth in ChatGPT, Claude, Google AIO, Perplexity and Gemini
Bot access counts in all five AI models, and what changes is each one’s path to the page.
| item | ChatGPT | Claude | Google AIO | Perplexity | Gemini |
|---|---|---|---|---|---|
| bot allowed | OAI-SearchBot, by OpenAI | Claude-SearchBot, by Anthropic; respected the block in the 2026 study | Googlebot rules, by Google | PerplexityBot and IP ranges, per Perplexity’s documentation | Google-Extended for the app; opened blocked pages in the 2026 study |
| text in the HTML | did not run JavaScript in 2024 (Vercel) | ClaudeBot did not run JavaScript in 2024 (Vercel) | Google processes JavaScript that is not blocked | did not run JavaScript in 2024 (Vercel) | Vercel wrote in 2024 that Gemini uses Googlebot’s infrastructure, which renders JavaScript |
| how the page is found | OpenAI’s search, with outside providers | Brave Search, according to MERJ and Profound | Google’s index | Perplexity’s own index, according to its documentation | Google’s index, in the app’s answers |
Allowing the bot counts for all five AI models. Text ready in the HTML was a condition for the OpenAI bots, for ClaudeBot and for PerplexityBot in the 2024 measurement, and the 2026 report about the OpenAI bot is in the guide about text in the HTML. In Google AIO, text in the HTML is more of a comfort than a condition, because Google writes that it processes JavaScript that is not blocked. Claude and Gemini part ways in robots.txt: in the 2026 study, Claude followed the block, and Gemini opened the blocked pages. The link between Claude and Brave Search, which Anthropic does not confirm, appears in the guide about how to get your company cited by AI models.
The impact of each change: what the sources show
The sources point to three blockers that can keep the page from reaching the AI model. The blocked bot is in the OpenAI and Anthropic rules, text outside the HTML in Vercel’s 2024 measurement, and the page outside the index in Google’s rule for AI Overview.
Two common things have no evidence in their favor. In a 14-day Otterly test, on Otterly’s own site, the markdown copy of two pages received no visits from AI bots. In Ahrefs’ calculation with 75 thousand brands, from December 2025, the number of inbound links bore little relation to the brand appearing in answers, and Ahrefs writes that correlation is not cause. Links from other sites are the subject of the guide about backlinks to appear in AI.
How to use the checklist on your company’s site
Use the checklist after seeing what AI models answer today, and go through the blocks in order, from access to the company being recognized. Take your buyer’s questions to ChatGPT, Claude and Google AIO and mark the answers in which the company appears. If it appears in none, start with access, text and the index. If it appears in part, look at the page that answers each question where it is missing.
Note the item that failed, the date of the fix and what changed. Then ask the same questions again. How to set up that calculation is in the guide about how to measure brand presence in ChatGPT.
At murmur, the checklist is the starting point, and the work is the page that answers
At murmur, the checklist is the starting condition, and the work is in the page that answers the buyer. murmur works on top of what the client’s site already shows. The sector study decides which questions get a page, and the guides go out on a subdomain of the client’s brand. The main site stays with the client’s team: when the measurement shows something to adjust on it, that reaches the team as a suggestion.
At Mindo, the diagnostic of 29 May 2026 found the site blank for the bot, and the guide went to guia.mindo.com.br with the text ready in the HTML, fully live on 22 June. What changed afterward in ChatGPT’s answers is in the Mindo case study. To see which pages reach the answers in a market, most of murmur’s sector studies show the sites the AI models used as a source in a round of buying questions.
Frequently asked questions
Does llms.txt go in the checklist?
llms.txt stays out of the checklist, because the published data does not show an effect from it for appearing in ChatGPT. In May 2026, Ahrefs looked at 137,210 sites, and 97% of valid llms.txt files had no access in the month. Google Search does not need the file, in the words of Google’s guide to AI features, and, on the pages about the bots, neither OpenAI nor Anthropic says that they open llms.txt. In a controlled test by Cisco DevNet, Cisco’s area for developers, with 15 AI tools, ChatGPT and Claude among them, none requested llms.txt, in the July 2026 version of the study. When the platform creates the file without work, there is no reason to remove it. The format is in the guide about llms.txt.
Does the checklist apply the same way to ChatGPT, Gemini, Perplexity and Google AIO?
The checklist applies to all of them in bot access and in the page that answers, and it changes at the entry point. Google AIO and the answers in the Gemini app use the Google index. Perplexity has its own index, and Claude cites mostly pages from the top of Brave Search, according to MERJ and Profound.
Does site speed change citation?
No study gathered up to October 2026 measured the connection between site speed and citation by AI models. On a March 2025 page, Bing lists loading time last among the search parameters, which it says are in general order of importance. Keep the site light for the person who visits, without counting on an effect on the answers.
Can I do the checklist alone, without an agency?
Running the checklist itself is simple, but the best option is to do the work with a GEO agency such as murmur. Besides checking the items, murmur applies data engineering to the company’s situation: it thoroughly maps out what the company is and does, compares the brand with those of competitors, measures reliably and delivers results. Going through the list on your own, without data or measurement, can help the site, but the same result is hard to reach. The checklist does not write the page that answers the buyer, nor does it show whether the company started to appear. To check the items, the tools are free: the browser, Search Console and Bing Webmaster Tools, and the check is redone with every change of platform, hosting or robots.txt. On the main site, murmur’s help is the diagnostic and the suggestion to the client’s team, and the new pages go out on a subdomain of the brand.
How often should the checklist be reviewed?
Review the checklist with every change to the site, the hosting or the robots.txt, and with every new bot announced by the AI companies. The date and the sitemap are checked with every page published. AI answers ask for a different frequency, explained in the guide about how often to measure brand presence.
Where to start, and where murmur comes in
Start with what AI models answer today about your market, and then with the first item on the list that fails on your site. With your site address, murmur’s free diagnostic shows in less than two minutes, and without sign-up, the answers from ChatGPT, Claude, Perplexity and Google AIO to your market’s questions. Fix that first item, check the next one and ask the questions again after each fix.
Read also
- llms.txt: What It Is, What It Is For, and Whether It Works to Get Cited by AI
- How to write text to appear in ChatGPT: answer up front and numbers with sources
- How ChatGPT search works: from the question to the sources in the answer
- How to get your company to appear in ChatGPT: the step by step, with no registration or ads
- Why ChatGPT does not recommend your company, and what to show the boss
References
- Google Search Central, “How to write and submit a robots.txt file”, updated on 21 November 2025. https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt (accessed on 30 September 2026).
- OpenAI, “Overview of OpenAI Crawlers”, undated page. https://developers.openai.com/api/docs/bots (accessed on 1 October 2026).
- Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, Claude Help Center, 7 April 2026. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (accessed on 5 October 2026).
- Perplexity, “Perplexity Crawlers”, Perplexity documentation, undated page. https://docs.perplexity.ai/docs/resources/perplexity-crawlers (accessed on 6 October 2026).
- Google, “Recursos de IA e seu site”, Google Search Central, Portuguese version last updated on 31 December 2025. https://developers.google.com/search/docs/appearance/ai-features?hl=pt-br (accessed on 3 October 2026).
- Google, “Google’s common crawlers”, Google-Extended section, last updated on 14 July 2026. https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (accessed on 5 October 2026).
- Cloudflare, “Have it both ways: stay discoverable in search while disallowing AI training”, 15 September 2026. Fewer than 1% of Cloudflare sites block search bots, and 17% use some kind of training block. Cloudflare sells bot access control. https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers (accessed on 17 September 2026).
- seoClarity, “ChatGPT’s 3 Bots: What Their Crawl Patterns Reveal [Research]”, September 2026. That is 308.6 million bot hits on corporate sites across several sectors, from 01/02 to 30 June 2026, with the medians for the portfolio. seoClarity sells SEO and AI search monitoring tools. https://www.seoclarity.net/research/chatgpts-3-bots-crawl-patterns (accessed on 17 September 2026).
- Gabriel Lopez-Fonseca, David Rodriguez, Stefan Bechtold and Jose M. Del Alamo, “Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers”, arXiv 2607.14447, version 1 of 16 July 2026 and version 2 of 19 July 2026, not peer reviewed. Ten assistants, four robots.txt conditions and 200 attempts; the access numbers are in Table 4. https://arxiv.org/abs/2607.14447 (accessed on 8 October 2026).
- Giacomo Zecchini, Alice Alexandra Moore, Malte Ubl and Ryan Siddle, “The rise of the AI crawler”, Vercel, with MERJ, 17 December 2024. Vercel sells website hosting. https://vercel.com/blog/the-rise-of-the-ai-crawler (accessed on 30 September 2026).
- Otterly, Rick Tousseyn, “GEO Experiment: Your Image Alt Text is Built for SEO. AI Search Reads Right Past It”, updated on 21 April 2026. 120 rounds across five AI models, on a test domain; the version with the data only inside the image had no correct answers. Otterly sells AI monitoring. https://otterly.ai/blog/geo-experiment-image-metadata/ (accessed on 1 October 2026).
- Google, “Controle de uso na IA generativa da Pesquisa”, Search Console Help, with the note on the rollout to all sites on 31 August 2026. https://support.google.com/webmasters/answer/16908024?hl=pt-BR (accessed on 3 October 2026).
- sitemaps.org, “sitemaps.org - Protocol”. https://www.sitemaps.org/protocol.html (accessed on 30 September 2026).
- Google, “Criar e enviar um sitemap”, Google Search Central, Portuguese version last updated on 15 July 2026. https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap?hl=pt-br (accessed on 8 October 2026).
- Google, “Guia detalhado de como a Pesquisa Google funciona”, Google Search Central, Portuguese version last updated on 18 December 2025. https://developers.google.com/search/docs/fundamentals/how-search-works?hl=pt-br (accessed on 8 October 2026).
- Google, “Como especificar um canônico com rel=“canonical” e outros métodos”, Google Search Central, Portuguese version last updated on 15 July 2026. https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls?hl=pt-br (accessed on 8 October 2026).
- IndexNow.org, “FAQ”, URL submission section, with the list of participating search engines. https://www.indexnow.org/faq (accessed on 8 October 2026).
- Microsoft, “Bing Webmaster Guidelines”, Bing Webmaster Tools documentation, undated page. They ask: “Place essential information near the top of the URL” and “Avoid mixing unrelated concepts on a single URL”, and they say that “URLs are more likely to be selected for grounding queries and citations when content stands on its own”. https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a (accessed on 1 October 2026).
- searchVIU, “Schema Markup and AI in 2025: What ChatGPT, Claude, Perplexity & Gemini Really See”, 2 December 2025. Test of 30 October 2025 on a page of searchVIU’s own site, which sells SEO tools. https://www.searchviu.com/en/schema-markup-and-ai-in-2025-what-chatgpt-claude-perplexity-gemini-really-see/ (accessed on 1 October 2026).
- Google Search Central, “Influence your byline dates in Google Search”, updated on 10 December 2025. https://developers.google.com/search/docs/appearance/publication-dates (accessed on 1 October 2026).
- Google, “Otimizar seu site para recursos de IA generativa na Pesquisa Google”, Google Search Central, Portuguese version last updated on 14 July 2026, about JavaScript. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide?hl=pt-br (accessed on 3 October 2026).
- Google Search Central, “Optimizing your website for generative AI features on Google Search”, with the note about llms.txt added on 15 June 2026. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide (accessed on 30 September 2026).
- Rick Tousseyn, “GEO Experiment: Markdown vs. HTML, Which Format Do AI Crawlers Prefer?”, Otterly, updated on 1 April 2026. Two pairs of pages on Otterly’s own site, for 14 days. Otterly sells AI monitoring. https://otterly.ai/blog/geo-experiment-html-vs-markdown/ (accessed on 30 September 2026).
- Ahrefs, Louise Linehan, “Top Brand Visibility Factors in ChatGPT, AI Mode, and AI Overviews (75k Brands Studied)”, 12 December 2025. It is correlation, and Ahrefs writes that correlation is not causation; the sample is domains with a Domain Rating above 40. https://ahrefs.com/blog/ai-brand-visibility-correlations/ (accessed on 1 October 2026).
- Louise Linehan and Xibeijia Guan, “We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read”, Ahrefs, 15 June 2026. Access logs from 137,210 sites with traffic in May 2026, of which about 38 thousand had a valid llms.txt. https://ahrefs.com/blog/llmstxt-study/ (accessed on 30 September 2026).
- Borysenko, “Developer Experience with AI Coding Agents: HTTP Behavioral Signatures in Documentation Portals”, Cisco DevNet, arXiv 2604.02544, version of 24 July 2026. Controlled test on a documentation portal, with 15 AI tools, among them ChatGPT and Claude: “No tool requested llms.txt”. https://arxiv.org/abs/2604.02544 (accessed on 17 September 2026).
- Microsoft, “How Bing delivers search results”, Microsoft Support, last updated in March 2025. The parameters come “in general order of importance”, and load time is the last one. Microsoft owns Bing. https://support.microsoft.com/en-us/bing/how-bing-delivers-search-results (accessed on 8 October 2026).
- Ryan Siddle, from MERJ, and Josh Blyskal, from Profound, “How Brave Search discovers new pages: A deep dive into the Web Discovery Project”, MERJ, 9 July 2026. About 35 thousand Claude citations; MERJ writes that Anthropic has not confirmed the connection with Brave. https://www.merj.com/blog/how-brave-search-discovers-new-pages (accessed on 3 October 2026).
- OpenAI, “Searching the web with ChatGPT”, OpenAI Help Center: the page links to Microsoft’s privacy policy and Shopify’s as search providers, without saying which questions each one handles. https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt (accessed on 30 September 2026).
- murmur: Mindo case study, https://murmur.marketing/en/cases/mindo/; free diagnostic, https://murmur.marketing/en/diagnostic/; sector studies, https://murmur.marketing/estudos/.