What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects data for training OpenAI models. OAI-SearchBot retrieves pages for ChatGPT Search so they can be surfaced and cited. Blocking GPTBot does not remove you from ChatGPT Search.

ClaudeBot is the Anthropic training crawler. Anthropic separately documents Claude-SearchBot for search indexing and Claude-User for user initiated fetches.

Is Google-Extended a crawler?

No. Google-Extended is a robots control token that expresses a preference about Google AI use of your content. There is no Google-Extended crawler to probe, so it can only be evaluated through robots.txt.

Which AI crawlers decide whether I can be cited?

The retrieval crawlers: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot and ChatGPT-User. A page that was never retrieved cannot be cited.

Reference

Every AI crawler, explained

Nine names appear in the access section of an AI Site View report. Here is what each one is, who runs it, and what it actually costs you to block it.

The distinction that matters most. Some of these fetch your page to answer a question now, which is what decides whether you can be cited. Others collect data to train a model. Blocking a training crawler is a policy choice with no effect on whether ChatGPT Search can reach you.

The matrix

Crawler	Job	How we test it	Scored
OAI-SearchBot OpenAI	ChatGPT Search retrieval	Live probe	9 points
PerplexityBot Perplexity	Perplexity index	Live probe	8 points
Googlebot Google	Search and AI Overviews grounding	Live probe	8 points
Claude-SearchBot Anthropic	Claude search indexing	Live probe	7 points
Bingbot Microsoft	Bing and Copilot grounding	Live probe	6 points
ChatGPT-User OpenAI	Live fetch when a user asks	Live probe	4 points
GPTBot OpenAI	Model training data	Live probe	Informational
ClaudeBot Anthropic	Model training data	Live probe	Informational
Google-Extended Google	Robots control token	robots.txt only	Informational

The six scored crawlers carry 42 points between them, inside the 60 point Technical Access half of the AI Site View Score. See the methodology for how the rest is composed.

The retrieval crawlers

OAI-SearchBot, OpenAI

Fetches pages for ChatGPT Search. If you care about being surfaced or cited in ChatGPT, this is the crawler that decides it. It is weighted highest for that reason.

PerplexityBot, Perplexity

Builds the index behind Perplexity answers. Perplexity cites sources visibly and sends real referral clicks, which makes it disproportionately valuable for a small site.

Googlebot, Google

The same crawler that has always powered Google Search, and the retrieval path that grounds AI Overviews. Blocking it is a decision about search itself, not only about AI.

Claude-SearchBot, Anthropic

Builds the index Claude searches. Worth knowing: Anthropic publishes the robots token and the role of each of its bots, but not the full user agent header string. The token is the part that robots.txt and server rules match on, and that part is Anthropic's own. Our probe header is corroborated by third party sources, and we say so rather than implying more certainty than we have.

Bingbot, Microsoft

Bing's index also grounds Microsoft Copilot, so this one crawler covers two surfaces. It is frequently forgotten in AI discussions and quietly matters.

ChatGPT-User, OpenAI

Not a background crawler. This arrives when a person asks ChatGPT to look at a specific page. Blocking it means someone who explicitly asked about your site gets nothing.

The training crawlers

These collect data used to train models. Allowing them can mean being described from a model's memory without a live fetch and usually without a link. Blocking them is a policy choice about how your content is used, and AI Site View reports it as a preference rather than a fault. Your score does not move either way.

GPTBot, OpenAI

Collects training data for OpenAI models. Blocking it does not remove you from ChatGPT Search, which is the single most common misunderstanding in this whole area. The decision framework is on should you block GPTBot.

ClaudeBot, Anthropic

Anthropic's training crawler, distinct from Claude-SearchBot which does search indexing, and from Claude-User which fetches a page when a person asks Claude about it.

The token that is not a crawler

Google-Extended, Google

Google-Extended is a robots control token, not a crawler identity we can meaningfully impersonate through an HTTP probe. No request ever arrives carrying it, so there is nothing to test at the door. We evaluate it through robots.txt policy alone and say so, rather than inventing a user agent and presenting the result as a live test.

What a probe can and cannot prove

Every live result on an AI Site View report comes from a request sent with the crawler's published user agent, from our own infrastructure. That is enough to reveal a robots rule or a firewall refusing that identity, which is where most real failures are. It is not proof that the genuine crawler receives the same response, because IP allowlists, reverse DNS validation and CDN logic can treat them differently. So the verdict is worded as what it is: no access barrier detected, at the time of the scan, rather than a promise about a crawler we do not operate.

See which of these your site actually answers. The free scan probes each crawler, cross checks robots.txt, and shows the evidence behind every verdict.

Check my site

Related reading: should you block GPTBot, how AI reads websites, and the measurement methodology.