Should you block GPTBot?
Most of the argument about this question is about content and consent. That part is yours to decide. This page is about the part almost nobody gets right: which crawler you are actually blocking, and what it costs you.
Two jobs, not one
Every AI crawler that reaches your site is doing one of two things, and the distinction decides everything that follows.
Training crawlers
They collect text used to train a model. Being in the training data can mean being described from memory, without a live fetch and usually without a link. Blocking them is a decision about how your content is used.
GPTBot (OpenAI), ClaudeBot (Anthropic), and Google-Extended, which is a robots control token rather than a crawler.
Retrieval crawlers
They fetch your page to answer a question now, and they are what decides whether you can be quoted or cited in an answer. Block these and you are not in the answer, whatever else you have done.
OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, ChatGPT-User.
This is why AI Site View scores only the six retrieval crawlers, which carry 42 of the 60 points in the Technical Access half of the score. GPTBot and ClaudeBot are still probed and still reported, as a training access preference marked allowed, blocked or unknown, and they earn no points in either direction. Blocking model training is a policy choice, not a defect, and we will not score it as one. The full weighting is on the methodology page.
The mistake to avoid
The common failure is a robots.txt that blocks GPTBot in the belief that this protects the site from AI, while OAI-SearchBot is also caught by a broad rule or a wildcard. The owner keeps their training stance and quietly loses the citation path as well. The opposite mistake is allowing everything because it all looked like one decision. It never was.
Three defensible positions
1. Allow retrieval, block training
The common choice for a business that wants to be found and cited but does not want its writing used to train a model it gets nothing from.
# Retrieval: allowed, so you can be fetched and cited
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# Training: blocked, which does not affect retrieval above
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /2. Allow everything
Reasonable if your goal is reach and you would rather be described from memory than not described at all. You do not need rules for this. An absent robots.txt rule is permission, so the honest version is simply not blocking anything.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml3. Block both
A real position for licensed archives, membership content and publishers with their own commercial arrangements. Just take it deliberately: this removes the citation path too.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /What robots.txt does not do
Robots.txt is a policy statement. Well behaved crawlers respect it, and the ones named here publish that they do. It is not enforcement, and more importantly for most sites, it is not the only thing deciding what a crawler receives. Your CDN, firewall or bot protection can refuse a crawler with a 403 while your robots.txt cheerfully welcomes it. That gap is invisible to any checker that reads the file and stops, and it is the most common real failure we see. There is a whole page on how that happens and how to fix it.
Rules are also matched per user agent, not inherited. A group for User-agent: * does not apply to a crawler that has its own group, so a specific block or allow needs to be written out for each token you care about.
Related reading: every AI crawler explained, how AI reads websites, and the measurement methodology.