TL;DR
Track AI crawlers in your server or CDN logs, not your analytics: filter requests by user agent, verify against published IP ranges, and group bots into training crawlers, search-index crawlers and on-demand fetchers. On otterly.ai, 61% of 67,238 AI agent visits in 30 days were on-demand fetches triggered by user prompts, only 7% were training crawlers, and the pricing page got more than a third of all visits.
Key takeaways
- AI bots don't show up in tag-based analytics. Use server or CDN logs.
- Group bots by purpose: training, search index and on-demand fetch mean very different things.
- On-demand fetches are the signal: a user's prompt sent an assistant to your page.
- Your pricing page is read live by assistants. On otterly.ai it got 36% of all AI agent visits.
- Block GPTBot if you like, not OAI-SearchBot: the settings are independent.
The short answer: how to track AI crawler visits
You track AI crawlers in your server or CDN logs, not in your web analytics. Bots like GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User and PerplexityBot don't run JavaScript, so tag-based analytics never sees them. Every request they make does show up in your access logs, with a user-agent string that names the bot.
- Get the raw logs from your web server, host or CDN (most CDNs can stream or export them).
- Filter by user agent against a current list of AI agents, and verify the big ones against the IP ranges their operators publish.
- Group the bots by purpose: training crawlers, search-index crawlers and on-demand fetchers. This is the step most people skip, and it's the one that matters.
- Count visits per page and per engine over at least four weeks, and compare windows rather than days.
You can do this with a script on a log export. The easiest way to track AI crawlers and agents is OtterlyAI Agent Analytics: add a lightweight integration at your server, edge or CDN, and it classifies the bots, agents and AI-referred visitors hitting your pages, by name and by URL.
Not all AI bots are the same: the three types
The single most useful thing I've learned from looking at our own logs: "AI crawler" is three different things, and lumping them together hides the signal.
| Type | What it does | Examples (from the operators' own docs) | What a visit tells you |
|---|---|---|---|
| Training crawler | Collects content that may be used to train AI models | GPTBot (OpenAI), ClaudeBot (Anthropic); Google-Extended is a robots.txt control token rather than a separate crawler | Your content may end up in future models. Says little about today's answers. |
| Search-index crawler | Builds the index an AI assistant searches when it answers | OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot | You're eligible to be found and cited in AI search answers. |
| On-demand fetcher | Opens a page in real time because a user's prompt needs it | ChatGPT-User, Claude-User, Perplexity-User | A real question, right now, led an assistant to your page. This is the closest thing to "AI traffic" you can see in logs. |
OpenAI, Anthropic and Perplexity all document these bots separately, and each one can be allowed or blocked on its own in robots.txt. One caveat from their docs: user-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt rules, because a person asked for the page.
What our own logs show: 67,238 AI agent visits in 30 days
Here's the same analysis run on us. At OtterlyAI we connect our own server logs to OtterlyAI Agent Analytics, exactly like customers do. For otterly.ai, 17 October – 15 November 2026:
Three things surprised me:
- Most AI bot traffic is people, not training. 41,278 visits (61%) came from on-demand fetchers, 21,077 (31%) from search-index crawlers and only 4,853 (7%) from training crawlers. If you block "AI bots" wholesale because you're worried about training, you mostly block buyers asking assistants about you.
- The pricing page is the most-fetched page, by far. /pricing/ got 24,080 agent visits, more than a third of the total, and 20,186 of those came from Claude. When someone asks an assistant "how much does X cost?" or "which tool fits my budget?", the assistant goes and reads your pricing page live. If that page is vague, gated or rendered only in JavaScript, the answer about you will be too.
- Engines behave very differently. Claude accounted for 33,855 visits (mostly Claude-User), Microsoft Copilot's search crawler for 15,229 and ChatGPT for 11,249 (mostly ChatGPT-User). Google showed up with just 518 visits, because Google's AI features largely rely on its regular search crawling, which isn't counted as an AI agent here.
A brand-new site looks completely different
For contrast, I connected this site, thomas-peham.com, after its relaunch. In its first two days of tracking (14–15 November 2026) it got 772 AI agent visits across 402 URLs. But the mix is the mirror image of otterly.ai: 51% training crawlers, 29% search-index crawlers and only 20% on-demand fetchers.
That's what a new or little-known site looks like: bots are discovering and indexing it, but few user prompts send assistants there yet. The pages they read most were my GEO explainer, robots.txt, the homepage and my piece on SEO for ChatGPT.
The lesson: raw visit counts don't tell you whether AI engines use your content. Watch the on-demand share. When it grows, assistants are actively pulling your pages into answers. Two days is a tiny window, so I'll update this section with four weeks of data.
How to block GPTBot (and when you shouldn't)
If you want to keep your content out of OpenAI's model training but stay visible in ChatGPT search, block the training crawler and allow the search crawler. OpenAI's documentation says each setting is independent:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
The same pattern works for Anthropic: ClaudeBot is the training crawler, while Claude-SearchBot and Claude-User power search and user requests. Anthropic notes that blocking those two may reduce your visibility in Claude's answers.
My take as someone who sells AI visibility: for most B2B companies, blocking training crawlers is a legitimate choice, but blocking search crawlers and on-demand fetchers is almost always a mistake. Those are the bots that put you in answers today. Also check your firewall or bot protection: a "block AI bots" toggle can block the good ones too, and robots.txt won't help if the request never reaches your server.
Which pages AI bots read, and what to do about it
- Pricing, product and comparison pages get fetched live during buying conversations. Keep them crawlable, in plain HTML, with prices and plan limits as text, not images.
- robots.txt and sitemap.xml are among the most-requested files on both of our sites. Keep them fast, valid and up to date.
- Explainers and guides are what search-index crawlers return to. An answer-first structure makes them easier to cite.
- Pages bots never visit are the real gap. If a page that should rank in AI answers gets zero agent visits in four weeks, check internal links, your sitemap and whether a firewall is blocking it.
Crawling is the precondition, not the result. Pair your log data with prompt tracking to see whether those visits turn into mentions and citations. I explain how we report that in the monthly AI search report I want as a CEO.
How to set it up
The do-it-yourself route: export a few weeks of access logs, filter on a maintained list of AI user agents, verify the big ones against the IP ranges their operators publish, then group by type, engine and URL. A spreadsheet works for a one-off audit.
If you want this ongoing, an agent analytics tool reads your logs continuously, keeps the bot list current and does the grouping for you. My checklist for choosing a GEO tool covers what to ask for.
Disclosure, and how we do it in OtterlyAI
I'm the CEO of OtterlyAI, so weigh this accordingly. OtterlyAI's Agent Analytics reads your server logs (file upload, webhook, Cloudflare, Netlify or a WordPress plugin), groups every AI agent into on-demand fetchers, search-index crawlers and training crawlers, and shows visits per page and engine. IP addresses are used only for de-duplication and aren't stored. It's available on the Standard, Premium and Custom plans.
Sources
- OpenAI, Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (ClaudeBot, Claude-User, Claude-SearchBot)
- Perplexity, Perplexity crawlers (PerplexityBot, Perplexity-User)
- Google, Google's common crawlers (Google-Extended)
- OtterlyAI Help Center, Crawlers in Agent Analytics (agent categories)
- OtterlyAI Agent Analytics data for otterly.ai (17 Oct – 15 Nov 2026) and thomas-peham.com (14–15 Nov 2026), pulled 16 Nov 2026.
Related reading: SEO for ChatGPT, how to track brand mentions in AI search and what GEO is.
FAQs
How do I track how often AI crawlers visit my website?
Use your server or CDN access logs. Filter requests by AI user agents such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User and PerplexityBot, verify them against the IP ranges their operators publish, and count visits per page and engine over at least four weeks. Tag-based analytics can't see these bots because they don't run JavaScript. The easiest way to track AI crawlers and agents is OtterlyAI Agent Analytics, which classifies those visits for you from a lightweight integration at your server, edge or CDN.
What is GPTBot?
GPTBot is OpenAI's web crawler for collecting content that may be used to train its generative AI models. It is separate from OAI-SearchBot, which surfaces websites in ChatGPT search, and from ChatGPT-User, which fetches pages when a user's request needs them.
How do I block GPTBot with robots.txt?
Add "User-agent: GPTBot" followed by "Disallow: /" to your robots.txt. To stay visible in ChatGPT search, keep OAI-SearchBot allowed; OpenAI treats the two settings independently.
Does blocking AI crawlers hurt my AI search visibility?
Blocking training crawlers like GPTBot or ClaudeBot doesn't remove you from AI search on its own. Blocking search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) or user-triggered fetchers can make it harder for assistants to find and cite your pages.
What is the difference between ChatGPT-User and OAI-SearchBot?
OAI-SearchBot crawls automatically to build the index behind ChatGPT search. ChatGPT-User visits a page only when a user's request in ChatGPT or a custom GPT needs it, so OpenAI notes robots.txt rules may not apply to it.
Do AI crawler visits mean my brand gets mentioned in AI answers?
No. Crawling is a precondition, not a result. Combine log data with prompt tracking to see whether AI engines actually mention and cite you for the questions your buyers ask.