Is Cloudflare hiding your site from AI?

Two of our own sites were answering 403 to every AI crawler for weeks. 22,143 pages, invisible. Their robots.txt explicitly welcomed all of them, and eight free checkers said both sites were healthy. This is how to find the same thing on your site, in two requests, without any tool.

---

The symptom: fine on Google, absent from AI answers

This failure is quiet by design, because nothing that reports to you is broken. The pattern looks like this.

• Search traffic is normal. Google Search Console shows impressions, clicks and indexed pages. Analytics looks like it always did.
• The site is perfect in your browser. Every page loads for you, for your client, and for anyone you ask to check.
• Assistants never mention you. Ask ChatGPT, Perplexity or Copilot about the thing you are the obvious answer to, and you are simply not in the reply.
• Your robots.txt is generous. You wrote it yourself. It welcomes everything by name.
• Cloudflare shows you nothing useful. You look in security events for a block, find no rule of yours firing, and conclude there is no block.

The reason the dashboard feels reassuring is that a mitigation issued by a managed feature is not the same object as a firewall rule you wrote, and it does not necessarily appear where you go looking for one. The response headers are the reliable witness. They come from the edge itself and they are not a summary of anything.

What we found on two of our own sites

This product exists because of one afternoon of measurement. Two directory sites we run, raretrial.org and areaiq.org, were returning 403 to every AI crawler we probed. Together that is 22,143 pages that no assistant could retrieve. Both had robots.txt files that named the crawlers and allowed them.

The cause was Cloudflare Bot Fight Mode, which is a separate setting from the AI Crawl Control toggles that had already been set correctly. It was issuing cf-mitigated: challenge above the origin, so nothing on the server was involved and nothing in the repository could have fixed it. Editing files changes what your server would have said. It does not change what the edge says instead.

Both zones were corrected. Rescanned afterwards, raretrial.org scored 100 and areaiq.org 92 under our published methodology. The sites were not improved in between. The barrier was removed.

One thing we will not tell you, because we could not verify it: there is a widely repeated claim that this platform switches on a block against AI crawlers for everybody, on a date. Our own measurements did not support the looser version of it. Bot-protection blocks are per-site settings. Check your own zone rather than trusting either a rumour or a supposed default.

Why your robots.txt is not the answer, and why eight checkers missed it

Eight free AI-readiness checkers looked at those two sites and reported them as healthy. None of them was broken. They were all doing the same limited thing: fetching /robots.txt, parsing it, seeing that AI crawlers were permitted, and reporting a pass. Not one of them made a request as a crawler, so a block sitting above the file was invisible to every one of them.

There is a second trap in the same file. Some CDNs inject a managed robots.txt block above yours, between markers such as # BEGIN Cloudflare Managed content and # END Cloudflare Managed Content. If that is happening, the file in your repository is not the file the internet is reading. The only way to know what your robots.txt says is to request it over the network, which is the third command below.

For the wider sequence of what happens when an assistant reaches a site, and the four places it usually goes wrong, see how AI reads websites. This page is about the very first step in that sequence, the one almost nobody checks.

Check it yourself with two requests

The test is a comparison. Ask for the same page twice, once identifying as a crawler and once identifying as a browser, then compare the two responses. If they differ, the difference is the answer. Replace yourwebsite.com in all of the commands below.

On Windows PowerShell, write curl.exe rather than curl, because curl there is an alias for a different command that does not take these flags.

1. The crawler request

This sends Googlebot's published user agent, discards the body and prints only the response headers.

curl -sS -o /dev/null -D - "https://yourwebsite.com/" \
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Run it again for each crawler that matters to you. These are the published user agents, exactly as we send them:

OAI-SearchBot     Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
PerplexityBot     Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Claude-SearchBot  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com)
Bingbot           Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
ChatGPT-User      Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
ClaudeBot         Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

A note on identity: sending one of these strings is a claim, not proof, and operators verify their real crawlers by network address as well. That is a strength for this test rather than a weakness. If a blunt rule blocks anything carrying the name, it blocks the real one too.

2. The browser control request

The same URL, with a realistic browser header set. This is the control, and it matters that it is realistic: header-poor requests are exactly what cheap bot filters reject, which would leave you unable to tell a crawler-specific block from a general one.

curl -sS -o /dev/null -D - "https://yourwebsite.com/" \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
  -H "Accept-Language: en-GB,en;q=0.9"

3. Your robots.txt as the internet actually reads it

curl -s "https://yourwebsite.com/robots.txt"

Read the whole output, not the part you wrote. If there is a managed block above your rules, this is where it appears, and editing your repository will not remove it.

Reading the results

A server: cloudflare line and a cf-ray header tell you the request went through that edge. A cf-mitigated header tells you the edge decided the outcome. Together they locate the problem precisely: above your origin, in your zone's settings.

Where the settings actually live

Everything below is per zone, which means per domain. Select the domain in the Cloudflare dashboard first, then work through these in order. Dashboard labels move between plans and redesigns, so treat the names as what to search for rather than a fixed path.

Bot Fight Mode, the one that caught us

Security > Bots. Bot Fight Mode issues a managed challenge to traffic it judges to be automated. Named search and AI crawlers are automated traffic, which is the entire point of them, so this can quietly apply to Googlebot and to every AI crawler at once. If your goal is to be retrievable, this switch works against you.

On Pro and above the equivalent is Super Bot Fight Mode, which is more precise. Set Definitely automated to Allow rather than switching the whole feature off, and leave the verified-bot allowance on. That keeps protection against the traffic you actually want to stop while letting named crawlers through.

Protect the things that genuinely need protecting with a path-scoped rule on your form and API endpoints, rather than one blunt site-wide switch that also happens to hide your content.

AI Crawl Control and the AI scraper toggles

A different part of the dashboard, usually under the AI or security sections depending on your plan and the current layout. This is where a managed robots.txt policy and AI-scraper blocking are controlled, and it is the setting most people find first because it has AI in the name.

Decide deliberately here, because two different questions get bundled together. Retrieval crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch pages so an assistant can answer a question about you now. Training crawlers such as GPTBot and ClaudeBot collect data for model training. Blocking training is a legitimate business choice and we never score a site down for it. Blocking retrieval is what makes you invisible. If a single toggle does both, you are choosing more than you think you are.

Three more places a block hides

• WAF custom rules. Security > WAF. Look for any rule with a Managed Challenge, Block or JS Challenge action whose expression matches on user agent, on cf.client.bot, or on a country or network. A rule written years ago to stop one scraper often catches everything named like a bot.
• Security Level. Set to High or Under Attack, this challenges far more traffic than most owners expect. Under Attack in particular is a temporary measure that frequently gets left on.
• Rate limiting. A crawler working through a large site looks like a burst of requests from one address. A tight limit turns a healthy crawl into a wall of 429 responses partway through.

One more distinction worth making before you change anything. A 403 to ClaudeBot or GPTBot alone, with retrieval crawlers answered normally, is probably a deliberate policy about model training and may be exactly what you intended. A 403 that also hits Googlebot and OAI-SearchBot is not a policy. It is a blunt bot rule, and it is costing you.

Why turning one off does not turn the other off

This is the specific thing that cost us weeks, so it is worth stating plainly. The AI crawler controls and Bot Fight Mode are separate settings, in separate places, acting on different criteria. Setting the AI controls correctly does not touch Bot Fight Mode. Bot Fight Mode does not read your AI preferences before deciding what looks automated to it.

So the reassuring sequence goes like this. You find the AI setting. You set it correctly. You confirm your robots.txt is generous. You reasonably conclude the job is done. Meanwhile a different switch, in a different menu, under a name with no AI in it, is answering 403 to the same crawlers. Both of our sites were in exactly that state.

The only reliable way to know your configuration is finished is to stop reading settings and make the request.

Confirm the fix, do not assume it

After you change a setting, give it a minute or two to propagate, then run the same commands again and compare against what you saw before. You are looking for all four of these.

• Every crawler you care about now returns 200, and the body is your real HTML rather than an interstitial.
• No cf-mitigated header on any of those responses.
• Your robots.txt over the network matches the file you wrote, with no injected block above it.
• The browser control still returns 200, so you have not opened anything you did not mean to open.

Then keep checking. A setting switched on once can be switched on again, by a colleague, by a plan change, or by an incident response that never got reverted. This is a configuration failure, which means it can come back as quietly as it arrived.

Be clear about what a clean result does and does not mean. It means no access barrier detected for that user agent, at that moment, from where the request was made. It is a point-in-time observation, not a guarantee of inclusion, citation or ranking anywhere. What can be said with confidence is the negative case: a crawler answered 403 retrieved nothing at all.

If you are not on Cloudflare

The method transfers unchanged, because the method is the comparison, not the vendor. Any layer in front of your application can answer before your application does: another CDN, a managed firewall, a reverse proxy, a hosting provider's bot protection, a security plugin, or a rule in your own nginx or Apache configuration that matches on user agent.

Send the crawler request, send the browser control, compare. If they differ, walk outwards from your application through every layer in front of it until you find the one that answered. The response headers usually name it.

Common questions

Does Cloudflare block AI crawlers by default?

Not in what we have measured. The blocks we found were per-site settings that were switched on in those particular zones, not a platform-wide default. That is why the answer is never to assume: send a request with a crawler's published user agent and read the status code that comes back for your own domain.

Why do free AI crawler checkers say my site is fine?

Most of them parse robots.txt and stop there. Eight free checkers reported two of our own sites as healthy while both were returning 403 to every AI crawler. Robots.txt is a policy statement about what you permit. The HTTP response is behaviour. A checker that only reads the policy cannot see the behaviour.

Why does the 403 not appear in my Cloudflare security events?

The dashboard view is a summary, and a mitigation issued by a managed feature does not always surface where you would look for a firewall rule you wrote yourself. Do not argue with the log. Read the response headers instead: a cf-mitigated header alongside a 403 is the edge telling you directly that it answered rather than your origin.

Does turning off AI Crawl Control also turn off Bot Fight Mode?

No. They are separate controls in separate parts of the dashboard, and they act on different things. AI Crawl Control governs a managed robots.txt policy and AI scraper rules. Bot Fight Mode challenges traffic it judges to be automated, which includes named search and AI crawlers. Setting one correctly leaves the other exactly as it was.

If I unblock the crawlers, will ChatGPT or Perplexity cite my site?

Nobody can promise that, and we will not. Removing a 403 removes a barrier that makes retrieval impossible. Whether any assistant then chooses to mention you depends on things a crawl cannot observe. We report what we can measure and label the rest as unknown, which is set out in full in our methodology.

The fast way to prove it either way

Everything above is work you can do yourself in about ten minutes, and you should know how, because it is your infrastructure. The scan below runs the same comparison for you across nine crawler identities at once, with your robots.txt policy read separately, so a disagreement between what you permit and what actually happens shows up as a finding rather than as silence.

Prefer to keep reading first? How AI reads websites covers the whole sequence, and the methodology sets out every weight, every crawler role and every limit of the method. Or run the free scan and read the evidence behind each line.