peham

CEO notes · AI visibility diagnosis16 min read

Not showing up in AI search?

Six different problems all look like "we're not in the answer". Here is how I find the step where a brand drops out of ChatGPT, Perplexity and Google's AI answers, and what our audit data at OtterlyAI says about the most overlooked one: blocked AI bots.

TL;DR

If your brand isn't showing up in AI search, find the step where it drops out: access (robots.txt, firewall, JavaScript), index, retrieval (a page that answers the question), trust (third-party sources the engine cites), accuracy, or measurement. Check access first. In OtterlyAI's audits, about 35% of websites our team audited blocked at least one AI bot, usually at the server or firewall while robots.txt allowed it. For ChatGPT allow OAI-SearchBot, for Perplexity allow PerplexityBot (including in your firewall), and for Google's AI features make sure pages are indexed and eligible for snippets.

Key takeaways

  • Diagnose before you write. Six different failures all look like "we're not in the answer".
  • robots.txt isn't the whole story. In our sample, about 64% of the sites whose server refused an AI bot had a robots.txt that allowed it.
  • Search bots and training bots are separate decisions. You can block GPTBot and still allow OAI-SearchBot.
  • Most AI crawlers don't run JavaScript. Put your key facts in the HTML.
  • For category questions, engines mostly cite third parties. If you're not on those pages, more of your own content won't fix it.

If your brand isn't showing up in ChatGPT, Perplexity, Gemini or Google's AI Overviews, it has almost always dropped out at one of six steps: the AI bots can't reach your pages, your pages aren't in the index the engine searches, no page of yours answers the question being asked, the engine trusts other sources about your category, the engine mentions you but gets you wrong, or you're judging a noisy channel from a handful of manual checks. Work through them in that order, because each step depends on the one before it:

  1. Access. Can the AI search bots actually fetch your pages? Check robots.txt, but also your firewall or CDN, and whether your main content is in the HTML without JavaScript.
  2. Index. Are your pages in the index each engine searches? For Google's AI features that's the normal Google index. ChatGPT and Perplexity run their own search crawlers.
  3. Retrieval. Do you have a page that answers the exact question a buyer asks, in plain text near the top?
  4. Trust. When the engine looks for sources about your category, do the pages it trusts (comparison articles, communities, review sites, docs) include you?
  5. Accuracy. If you are mentioned, is the description right and current?
  6. Measurement. Are you tracking a fixed set of prompts across engines over weeks, or typing three questions into ChatGPT once?

Most teams jump straight to step 3 and write more content. In my experience the cheapest wins are usually at step 1, and the most common real cause is step 4. Below is how to check each step, what our own audit data at OtterlyAI shows, and what to do on each engine.

Where brands fall out of AI answers: six steps, six fixes

An AI search answer is built in a chain: crawl, index, retrieve, pick sources, write the answer, mention brands, cite links. You can fail at any link, and each failure looks the same from the outside: "we're not there". The symptom you see tells you where to look.

StepTypical symptomHow to checkFix
1. AccessAbsent everywhere, even for your own brand name, or present only in Google's AI featuresrobots.txt rules for the search bots; request a page with each bot's user agent; view the raw HTML without JavaScriptAllow the search and user-triggered bots; add a firewall/CDN allow rule for their published IP ranges; render main content on the server
2. IndexNew pages never cited; Google AI features ignore a pageIs the page indexed in Google with a snippet? Do AI search bots visit it in your logs?Fix indexing and noindex/nosnippet settings; link new pages from pages that are already crawled; keep a clean sitemap
3. RetrievalCited for your brand name, absent for category questionsWhich buyer questions do you have a page for? Is the answer in the first paragraph?One page per important question, answer first, with specific facts (prices, limits, integrations) in text, not images
4. TrustCompetitors named, you're not; the cited pages are third-party articlesOpen the citations for your target prompts. Are you on those pages?Earn accurate inclusion on the pages engines already cite; contribute to the communities and publications they draw on
5. AccuracyMentioned, but with old pricing, the wrong category or a mix-up with another companyCompare the wording of the answer with your own pages and profilesCorrect your own sources first, then the third-party ones (my playbook for wrong AI answers)
6. Measurement"We were in it last week and now we're gone"How many prompts, engines and days is the conclusion based on?Track a fixed prompt set daily across engines and compare equal multi-week windows

One pattern worth knowing: if you appear in Google's AI Overviews or AI Mode but nowhere else, step 1 is the prime suspect. Google's AI features use Google's normal crawler and index, while ChatGPT, Perplexity and Claude rely on their own bots, which firewalls block far more often.

What our audits show: about 35% of sites block an AI bot

At OtterlyAI, our Crawlability Checker requests a site's homepage with the user agents of 19 to 21 AI bots (search bots, user-triggered fetchers and training crawlers) and also reads the site's robots.txt. I looked at the most recent check for websites our team audited between January and September 2026. This is a small sample, not a representative study, but the pattern was hard to miss:

about 35%
of websites our team audited blocked at least one AI bot, through robots.txt, the server or both
about 64%
of sites whose server refused a bot had a robots.txt that allowed it
about 11%
of websites our team audited refused every AI bot we tested

The interesting number is the middle one. Only about 46% of the sites that blocked a bot did so in robots.txt, which is the place most marketers look. About 85% of them refused at least one bot at the server level, almost always with an HTTP 403, and for about 64% of those server-level blocks, robots.txt said "come in". In those cases the marketing team's robots.txt says one thing and the firewall, CDN or bot-protection setting does another. These server-level blocks often happen without anyone deciding on them: bot protection is set up by people whose job is to stop bad traffic, and an AI crawler looks a lot like bad traffic. Among the bots refused at server level were the ones that decide whether you can appear in AI search: PerplexityBot was refused at the server by about 16% of the websites our team audited, and ChatGPT's search and user bots (OAI-SearchBot and ChatGPT-User) by about 14%.

The second check is rendering. Our Content Audit compares what a page shows with and without JavaScript. Among third-party pages our team audited, about 15% showed half or less of their content without JavaScript, and about 13% showed essentially none of it. That matters because most AI crawlers don't run JavaScript: a 2024 analysis by Vercel found that none of the major AI crawlers it studied rendered JavaScript, with Google's Gemini (which uses Googlebot's infrastructure) as the notable exception.

A caveat, to be fair to those sites: our check sends each bot's user-agent string from our own servers. Some firewalls block anything that claims to be a bot but doesn't come from the bot's published IP ranges, and would let the real crawler through. So a 403 in our check is a strong reason to look, not proof that the real bot is blocked. Your server or CDN logs give you the definitive answer, as I explain in AI crawlers: how to track them.

How to diagnose AI search visibility problems, step by step

You can do a first pass of all six checks in an afternoon. Start with the questions where you expected to appear and didn't, and keep notes per engine, because the answer is often different on each one.

1. Check access: robots.txt, firewall and rendering

Open your robots.txt and look for rules that affect OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User and, for Google, Googlebot. Then check what the server does: ask whoever runs your CDN or firewall whether "bot fight", "AI scraper" or rate-limit rules are on, and look for 403 responses to those user agents in your logs. In July 2025, Cloudflare announced it was changing its default to block AI crawlers unless they pay, so many sites now block AI bots without anyone in marketing knowing. Finally, open a key page with JavaScript disabled, or view the raw HTML. If your product description, pricing or FAQ isn't there, most AI crawlers won't see it either.

2. Check the index

For Google's AI Overviews and AI Mode the rule is simple: a page must be indexed and eligible to be shown in Google Search with a snippet, and there are no additional technical requirements. A nosnippet or noindex tag removes it. For ChatGPT and Perplexity, look in your logs for visits from OAI-SearchBot and PerplexityBot to the page in question. A page their search bots have never fetched can't be cited from their index. New pages get found faster when they're linked from pages that are already crawled often.

3. Check retrieval: do you answer the actual question?

List the 10 to 20 questions buyers ask an assistant before they buy something like yours, and next to each one write the URL of your page that answers it. The gaps are usually obvious. AI engines tend to quote the paragraph that answers the question directly, so put the answer in the first two or three sentences and include the concrete facts in text: what it costs, who it's for, what it integrates with. My prompt research playbook covers how to pick the questions.

4. Check trust: who gets cited instead of you?

Run your target prompts and open every citation. For category questions ("best X for Y", "how do I solve Z"), the cited pages are mostly third-party: comparison articles, community threads, video, publisher guides and the engines' own documentation. On the "why isn't my brand showing up" questions we track, YouTube, Reddit, LinkedIn and Quora were among the most-cited domains, next to OpenAI's and Google's own documentation. If you're not on the pages the engine trusts, you won't be in the answer, however good your own site is. The fix is accurate, earned inclusion on those pages, not more pages on your own domain.

5. Check accuracy

If you're mentioned but described wrongly (old pricing, wrong category, confused with another company), that's a different problem with a different fix. Trace the wrong sentence back to its source and correct that source. I wrote a separate guide on how to stop AI search from getting your company wrong.

6. Check your measurement

AI answers change from run to run. In OtterlyAI's data, the citations of a typical top page moved by about 19% from one week to the next without anyone touching it (I describe this in GEO experiments). So "we were in it on Monday and gone on Wednesday" may be nothing at all. Track a fixed set of prompts daily, across the engines your buyers use, and compare multi-week windows before you conclude anything.

Why Perplexity isn't citing your website

Perplexity is the engine where access problems show up most clearly, because it relies on its own crawler and fetches pages live. Its bot documentation separates two agents: PerplexityBot, which surfaces and links websites in Perplexity's search results (and, Perplexity says, isn't used to train foundation models), and Perplexity-User, which visits a page when a user's question needs it. Perplexity recommends allowing PerplexityBot in robots.txt and allowing requests from its published IP ranges, and it gives explicit firewall instructions: if you use a WAF, you may need an allow rule that combines the user agent with those IP ranges.

So if Perplexity never cites you: check that PerplexityBot isn't disallowed in robots.txt, check your firewall logs for 403s to PerplexityBot and Perplexity-User, and make sure the answer is in the server-rendered HTML. In our audit sample, PerplexityBot was among the bots most often refused at server level. If access is fine, the problem is usually step 3 or 4: no page that answers the question, or the sources Perplexity trusts on the topic don't mention you.

Is your website discoverable on ChatGPT?

For ChatGPT search, the bot that matters is OAI-SearchBot. OpenAI's crawler documentation says that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers (they can still appear as navigational links), and recommends allowing both the bot in robots.txt and requests from its published IP ranges. It also says changes to robots.txt can take about 24 hours to take effect for search.

Three details that trip people up:

  • GPTBot is a different decision. GPTBot crawls content that may be used to train OpenAI's models. OpenAI says each setting is independent, so you can block GPTBot and still allow OAI-SearchBot. Blocking training doesn't have to mean disappearing from ChatGPT search.
  • ChatGPT-User doesn't decide search inclusion. It fetches pages when a user's request needs it, and OpenAI says it isn't used to determine whether content may appear in search.
  • Ranking in Bing or Google isn't the whole story anymore. ChatGPT search launched on third-party search providers, but OpenAI now documents that some ChatGPT versions can search "OpenAI's indexed and cached web content". My team at OtterlyAI wrote up the evidence for OpenAI's own search index. If OAI-SearchBot can't fetch you, good rankings elsewhere won't fully make up for it.

Even when the bot can reach you, inclusion isn't guaranteed. Then you're back at retrieval and trust: does a page of yours answer the question, and do the sources ChatGPT cites on the topic mention you?

Google AI Overviews and AI Mode: the classic SEO checks still apply

Google is the simplest case. Its AI features run on Googlebot and the normal Search index, so the checks are classic SEO ones: is the page indexed, is it eligible for a snippet, and is it the best answer for the query? Google says there are no additional requirements for appearing in AI Overviews or AI Mode, and its guidance on generative AI search adds that you don't need special schema markup or AI text files such as llms.txt. Also note that Google-Extended, the robots.txt token for limiting AI training and grounding in some of Google's other systems, isn't the control for Search. Googlebot is.

If you rank well in classic Google results but never appear in AI Overviews for the same queries, look at the answer itself: the overview usually draws on pages that answer the question more directly or more completely than yours.

My take as a CEO: invisibility is rarely a content-volume problem

When a brand isn't showing up in AI search, the reflex is to produce more content. I understand the reflex, and I think it's usually wrong.

The data I see points to two much less glamorous causes. The first is access. A surprising share of sites block the very bots that decide whether they can be cited, and in most cases nobody made that decision. A security setting did. That's a conversation between marketing and whoever owns the CDN, and it costs nothing. The second is trust. For category questions, AI engines mostly cite third parties. If you're absent from those pages, ten more blog posts on your own domain won't change the answer.

I'll also be honest about our own numbers. OtterlyAI tracks roughly 430 prompts about AI search for our own brand. On the seven questions this article is about ("why is my brand not showing in AI search results", "Perplexity not citing my website", "is my website discoverable on ChatGPT" and similar), OtterlyAI was mentioned in only about 2 of every 100 AI answers over two weeks in November 2026, and in none of the Perplexity or Copilot answers. That's exactly the gap this article is meant to close, and it's how I'd tell any team to pick its next piece of content: find the questions where you should be the answer and aren't, then work out at which step you fall out.

Three things I'd do this week, in this order: ask your CDN or security owner whether AI bots are blocked, and allow the search bots you want; check that your key pages show their main content without JavaScript; then pick the ten buyer questions that matter most and look at who gets cited instead of you.

How OtterlyAI helps with this diagnosis

Disclosure, and what OtterlyAI offers here

I'm the CEO of OtterlyAI, so weigh this accordingly. OtterlyAI's Crawlability Checker tests whether AI bots can access a website, checking both robots.txt and the server's response, and the Content Audit checks pages for AI-readiness factors, including what shows without JavaScript. Brand monitoring tracks a fixed set of prompts daily across ChatGPT, Google AI Overviews, Google AI Mode, Gemini, Perplexity, Microsoft Copilot and Claude, and records which brands are mentioned and which URLs are cited, so you can see at which step you fall out on each engine. Agent Analytics shows which AI crawlers and agents actually visit your pages. Plans start at $29/month, with a free trial.

Sources

Related reading: AI crawlers: how to track them, how to stop AI search from getting your company wrong, GEO experiments and SEO for ChatGPT.

FAQs

Why is my brand not showing in AI search results?

Usually because it drops out at one of six steps: AI bots can't access your pages (robots.txt, firewall or JavaScript rendering), your pages aren't in the index the engine searches, no page of yours answers the question, the engine trusts third-party sources that don't mention you, it mentions you incorrectly, or you're judging from a few manual checks. Check access first: in OtterlyAI's audits, about 35% of websites our team audited blocked at least one AI bot, often at the server level while robots.txt allowed it.

Why is Perplexity not citing my website?

First check that PerplexityBot (search) and Perplexity-User (live fetches) can reach your pages. Perplexity recommends allowing PerplexityBot in robots.txt and its published IP ranges in your firewall. Then make sure the answer is in the server-rendered HTML. If access is fine, you probably lack a page that answers the question directly, or the sources Perplexity trusts on the topic don't mention you.

Is my website discoverable on ChatGPT?

It can be, if OAI-SearchBot is allowed. OpenAI says sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers, except as navigational links. Allow it in robots.txt and in your firewall, and expect about 24 hours for robots.txt changes to take effect. Blocking GPTBot (training) is a separate setting and doesn't remove you from ChatGPT search.

Why are brand mentions missing from ChatGPT responses?

If ChatGPT can access your site but doesn't name you, the cause is usually retrieval or trust: there's no clear page answering the question, or the third-party pages ChatGPT cites for your category don't include you. Open the cited sources for the prompts you care about and work on accurate inclusion there.

How do I audit a website for answer engine optimisation?

Check six things in order: AI bot access (robots.txt plus server responses to each bot's user agent), rendering (is the main content in the HTML without JavaScript), indexing (Google index and snippet eligibility, AI search bot visits in your logs), answer coverage (a page per important buyer question, answer first), third-party presence on the sources engines cite, and accuracy of what engines say about you. Then track a fixed prompt set across engines to measure progress.

How long does it take to show up in AI search after fixing access?

OpenAI says robots.txt changes take about 24 hours to be reflected in ChatGPT search. Being crawled, indexed and then chosen as a source takes longer and varies by engine and page. Track the same prompts daily and compare multi-week windows, because AI answers vary a lot from week to week on their own.

Keep
reading.

All writing →
OtterlyAI

See how your brand shows up in AI search.

Track mentions, citations and share of voice across ChatGPT, Perplexity, Gemini, Google AI Overviews, AI Mode and Copilot.

Try OtterlyAI ↗

Talks · advisory · press

Want Thomas on your stage or podcast?

Get in touch →