How to check if ChatGPT can reach your website
There is a quick answer to this and a correct one. The quick answer is a robots.txt lookup, which is what almost every free checker hands you in two seconds. The correct one takes about ten minutes, needs nothing but curl, and finds the failures the quick answer cannot see.
"Can ChatGPT see my site" is four questions wearing one coat
The sentence feels like a single yes or no. It is not. It hides four independent conditions, and each one can pass while the next one quietly fails. A site is only genuinely reachable when all four hold at the same time.
1. Does robots.txt permit the retrieval crawlers?
The policy layer. It is a file you publish that tells well behaved crawlers where they may and may not go. Cheap to read, easy to get wrong, and the only thing most free tools look at.
2. Does your infrastructure actually serve them?
The behaviour layer. Your server, CDN, WAF and rate limiter all decide what to do with the request before your website is involved at all. This is where most real failures live, and nothing in your analytics reports it.
3. Is your content in the HTML the server returns?
The content layer. Many automated crawlers rely heavily on server-delivered HTML. If your page assembles itself in the browser, what the server sent may be an empty shell with a script tag in it.
4. Can a machine establish the basic facts from it?
The meaning layer. Reaching a page is not the same as being able to state who you are, what you sell, who it is for and what it costs. Most sites never say those things in a form a machine can lift.
Nearly every free AI visibility checker tests the first one and stops. That is not a conspiracy, it is economics: parsing a text file is nearly free, while sending real requests from real infrastructure and reading what comes back costs money on every single scan. So the cheap test became the standard test, and a whole category of tools now reports on policy while calling it visibility.
robots.txt is a statement of policy. The HTTP response is behaviour. When the two disagree, only a real request will ever find it.
Test it yourself, properly, in about ten minutes
Everything below runs in a terminal with curl, which ships with macOS, most Linux distributions and Windows 10 and later. Replace https://yourwebsite.com/ with your own address, including the trailing slash, and keep the protocol on the front.
Step 1. Establish a browser baseline
Start with what a normal visitor gets. Without this number, a crawler result means nothing, because you cannot tell a crawler block from a site that is simply down.
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" https://yourwebsite.com/
You want 200. If this returns anything else, fix that first: no crawler result you gather afterwards will mean what you think it means.
Step 2. Repeat it as a published crawler user agent
Now send the identical request and change exactly one thing, the user agent. This is the whole test. Any difference in the response is caused by the identity in that header and nothing else.
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://yourwebsite.com/
Browser returns 200 and this returns 403? You have found something real, and you found it in two commands.
Doing that once per crawler is tedious, so here is the whole panel in one script. Save it as probe.sh and run it with bash probe.sh.
SITE="https://yourwebsite.com/"
probe () {
printf '%-17s %s\n' "$1" "$(curl -s -o /dev/null -w '%{http_code}' -A "$2" --max-time 15 "$SITE")"
}
probe "Browser control" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
probe "OAI-SearchBot" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
probe "PerplexityBot" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
probe "Claude-SearchBot" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com)"
probe "Googlebot" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
probe "Bingbot" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
probe "ChatGPT-User" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
A healthy site prints 200 down the whole column. One odd number in that column is the entire point of running it.
Step 3. Read the response code properly
Status codes are the vocabulary of this test, and a couple of them mean something quite different from what people assume.
| Code | What it means here | What to do |
|---|---|---|
| 200 | Served. Probably. See step 4 before you celebrate. | Compare the response size against the browser baseline. |
| 301 308 | A redirect. Normal, and usually fine. | Add -L and check where you actually land. |
| 401 | An authentication wall in front of the page. | Often a staging password or preview protection left switched on. |
| 403 | Refused. The classic bot protection answer, and the one to take seriously. | Look at your CDN and WAF settings, not at your website code. |
| 404 | That URL is not there. | Check the address, including www and the trailing slash. |
| 429 | Rate limited. Intermittent, so it is often missed entirely. | Re-run a few times. A crawler that meets this repeatedly backs off. |
| 503 | Unavailable, and frequently a challenge or under attack mode rather than an outage. | Compare against the browser baseline. If only the crawler sees it, it is a rule. |
Step 4. Catch the 200 that is not a page
This is the trap that makes naive testing worse than no testing. A bot challenge page is served with status 200. It looks like success in every log and every dashboard. It contains a few kilobytes of JavaScript, a spinner, and none of your content.
Two cheap tells. First, size. Fetch the page as a crawler and count the bytes, then do the same with the browser user agent from step 1.
curl -s -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://yourwebsite.com/ | wc -c
If the browser gets 40,000 bytes and the crawler gets 3,000, you are not looking at your homepage.
Second, the headers. Print them without downloading the body.
curl -s -D - -o /dev/null -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://yourwebsite.com/ | head -20
Read what comes back. A header such as cf-mitigated: challenge is your CDN telling you plainly that it intercepted the request. Interstitials are also usually served with cache-control: no-store and no meaningful content-length.
Step 5. Read robots.txt over the network, never from your repository
The file in your project folder is not evidence. Some platforms and CDNs serve a managed robots.txt above your origin, which means the file you carefully edited is never the file anyone receives. Ask the network what it says.
curl -s https://yourwebsite.com/robots.txt
Two things catch people out. Rules are matched by specificity rather than by order, so a broad Disallow can be overridden by a longer Allow. And a group with no matching user agent line simply does not apply, so a carefully written block can be doing nothing at all.
Step 6. Confirm your content is in the HTML the server returned
Rendering behaviour differs between crawlers and changes over time, so the durable question is not whether a particular crawler runs JavaScript. It is whether your content exists in the HTML your server delivers, without browser execution. Many automated crawlers rely heavily on exactly that. Take a sentence you know is visible on the page and look for it.
curl -s -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://yourwebsite.com/ | grep -c "a sentence you know is on your page"
A result of 0 means that sentence is not in your server-delivered baseline. The browser view is not the machine view, and the gap between them is invisible until you look.
Step 7. Confirm the basic facts are machine readable
The final layer, and the one people skip. Reaching your page is worth nothing if nothing on it states, in a liftable form, who you are and what you do.
curl -s https://yourwebsite.com/ | grep -o "<title>[^<]*</title>" curl -s https://yourwebsite.com/ | grep -c "application/ld+json"
The first prints your title as a machine receives it. The second counts your Schema.org structured data blocks; 0 means every fact about your business has to be inferred from prose, and inference is where a machine quietly gets you wrong. Our methodology sets out exactly what we accept as an established fact and what stays UNKNOWN.
Which crawler identities actually decide citation
Not every crawler with AI in its name does the same job, and confusing the two kinds is the most common mistake in this whole area.
Retrieval crawlers: these are the ones to test
- OAI-SearchBot, OpenAI. Retrieval and indexing for ChatGPT Search.
- ChatGPT-User, OpenAI. The live fetch triggered when a person asks about you in a conversation.
- PerplexityBot, Perplexity. The Perplexity index.
- Claude-SearchBot, Anthropic. Claude search indexing.
- Googlebot, Google. Google Search, and the grounding behind AI Overviews.
- Bingbot, Microsoft. Bing, and the grounding behind Copilot.
Two of those are not AI branded at all, and that surprises people. Assistants that ground their answers in a conventional search index inherit that index's view of you, so classic search crawlers remain part of the answer to the question in the title.
Training crawlers: a policy choice, not a defect
GPTBot and ClaudeBot collect data for model training. They are a different question entirely.
This is why our scan reports training access as a neutral preference and never lets it move the score. If you want the longer version of who all of these are, the crawlers explained goes through them one at a time.
Why asking ChatGPT itself is not a test
This is the advice you will find almost everywhere: open ChatGPT, ask it about your company, see what it says. It is a reasonable instinct and it is not a measurement. Four reasons, and each one is enough on its own.
- It is not reproducible. The answer varies with the session, the exact phrasing, whether browsing was used on that turn, the model version, and defaults that change without announcement. Ask twice, get two answers, and you have no way to diff them.
- A confident answer does not mean anything was retrieved. An assistant can produce a fluent, plausible description of your business with no request to your site at all: from training data, from a directory listing, from a competitor's comparison page, or from general knowledge of your sector. You cannot tell those apart from the output.
- The failure direction is just as ambiguous. A vague or wrong answer does not prove you are blocked. It might mean the model did not browse on that turn, that the retrieval picked a different page, or simply that it answered from memory.
- It cannot tell you why. There is no status code to read, no robots decision to inspect, no timestamp, no record of which URL was fetched. Even when the answer is wrong in an interesting way, you learn nothing you can act on.
Compare that with step 2 above. One request, one changed header, one number, repeatable on demand and explainable to a developer in a sentence. That is the difference between a measurement and a story about one afternoon.
We do not query ChatGPT, Claude or Gemini as part of a scan, and we never report what one of them said about you. Not because it would be difficult, but because it would be unreproducible, and we would be selling you anecdote with a number printed on it.
The failure that started this product
Two sites, raretrial.org and areaiq.org, were returning 403 to every AI crawler that asked. 22,143 pages, invisible. Their robots.txt files explicitly welcomed all of them, in writing, and had done for months.
The cause was Cloudflare Bot Fight Mode, a security setting acting above the origin. Every human visitor got the site. Every named crawler got refused. Nothing in the analytics said a word about it, because a request that is turned away at the edge never reaches the analytics.
That is the whole argument for testing behaviour rather than policy, and it is why this page is written the way it is. If your site sits behind a CDN, read how Cloudflare settings block AI crawlers next, because the settings involved are not where most people expect to find them.
What no test can tell you, ours included
Everything above is a genuine improvement on a robots.txt lookup. It is still not omniscience, and any tool that implies otherwise is selling you certainty it does not have. Here is where the method stops.
- A probe is not the genuine crawler. A request sent from our infrastructure, or from your laptop, carrying a crawler's published user agent is not proof that the real crawler receives the same response. IP allowlists, reverse DNS validation and CDN rules can treat the two differently, in either direction. A probe result is evidence about a probe, and we word every result that way.
- Results are point in time. A security setting toggled tomorrow, a plugin update, a new firewall rule or a changed platform default can reverse any of this without anyone touching your website. That is precisely why these failures run for months.
- Permission is not inclusion. No barrier to a crawler does not promise retrieval, citation, ranking or a recommendation. It removes a reason to be excluded. Anyone promising the rest is guessing.
- Nobody outside those companies can tell you what an assistant privately concluded about your business. There is no API for that and no honest way to infer it from a crawl. On our reports it is labelled UNKNOWN, in those words, on purpose.
The four honest verdicts we will report for a crawler are: no access barrier detected, blocked by robots.txt, access barrier detected, or could not be determined during this scan. Only the robots.txt one is a statement of fact, because it is a policy you published. The others are observations, and they are worded as observations.
Questions people actually ask
Can ChatGPT read my website?
It depends on the four conditions at the top of this page, and each can pass while the next fails. The only way to know is to send real requests carrying the published user agents and compare them against a normal browser request.
My robots.txt allows AI crawlers. Does that mean they reach my site?
No. robots.txt states what you permit. The HTTP response is what happens. A bot protection rule can answer 403 to a named crawler while your robots.txt welcomes it and your homepage loads perfectly for people. That exact disagreement, across 22,143 pages, is why this product exists.
Does blocking GPTBot remove me from ChatGPT Search?
No. GPTBot is a model training crawler. Retrieval for ChatGPT Search is governed by OAI-SearchBot, and the fetch triggered inside a conversation by ChatGPT-User. Blocking training while allowing retrieval is a perfectly coherent position, and it is a policy choice rather than a fault.
Is asking ChatGPT about my website a reliable way to test it?
No, for the four reasons set out above: it is not reproducible, a confident answer does not prove retrieval, a poor answer does not prove a block, and it cannot tell you why. It is anecdote, not measurement.
How often should I re-check?
Every result is point in time, and the changes that break this are usually made by someone else: a security setting toggled at the CDN, a plugin update, a platform default that moves. Re-check after any hosting or infrastructure change, and on a schedule otherwise, because nothing in your analytics will ever report a crawler that was turned away at the door.
Or run all seven steps at once
Everything on this page, in about twenty seconds, on your homepage. The scan sends real requests with each crawler's published user agent, reads your live robots.txt separately, flags every place where the two disagree, checks what your server delivered without browser execution, and shows the evidence behind every single line: the probe identity, the status code, the matched robots rule and the timestamp. Free, and no signup.
We show what we measured, label what we inferred, and say plainly what we cannot know.
Related reading: how AI reads websites, when Cloudflare blocks AI crawlers, the AI crawlers explained, and the methodology behind every number we report.