Can AI crawlers actually reach your site?
Answer bots and training bots are two different decisions, and one robots.txt line can take you out of AI answers entirely. Audit them separately, and know what robots.txt cannot fix.
· · By the Askwords team
There is a version of the AI visibility problem that no amount of content will fix, and it takes ten minutes to rule out: the crawlers behind these answers cannot read your website.
It happens for reasonable-sounding reasons. Somebody read an article about AI companies training on the open web, opened robots.txt, and blocked everything that looked like an AI bot. The training opt-out worked. So did the accidental opt-out of every citation the business might have earned.
Training is a policy choice. Answering is a visibility choice.
AI crawlers do two unrelated jobs. Conflating them is the most common mistake in this whole area.
GPTBot, Google-Extended
Collect pages to train future models. Blocking them is a legitimate business and rights decision, and it does not remove you from today’s AI answers. Publishers block these all the time and stay perfectly visible.
OAI-SearchBot, PerplexityBot, Googlebot, Bingbot…
Fetch pages so an answer engine can answer with them, today, with a link. Blocking one of these does not protect anything. It removes you from that answer engine’s citations entirely.
Know which line does what
These are the user-agent tokens worth having an explicit opinion about. The first group gates whether AI can cite you at all. The second is where a considered opt-out belongs, if you want one.
Blocking these costs you citations
| User-agent | Belongs to | What it does |
|---|---|---|
| OAI-SearchBot | ChatGPT Search | Indexes pages for ChatGPT's answers: blocked means ChatGPT cannot cite you |
| ChatGPT-User | ChatGPT browsing | Fetches your pages live when a ChatGPT user asks about them |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers and citations |
| Claude-SearchBot | Claude Search | Indexes pages for Claude's search answers and citations |
| ClaudeBot | Claude | Anthropic's crawler behind Claude's web answers |
| Googlebot | Google + AI Overviews | AI Overviews and AI Mode can only cite pages Googlebot can crawl |
| Bingbot | Bing (feeds Copilot and ChatGPT) | Copilot answers from Bing's index, and it still feeds part of what ChatGPT retrieves alongside OAI-SearchBot: blocked here narrows both |
Blocking these is a policy choice, not a visibility one
| User-agent | Belongs to | What it does |
|---|---|---|
| GPTBot | OpenAI training | Training-data crawler: blocking it is a policy choice and does not affect ChatGPT Search |
| Google-Extended | Gemini training | Gemini training opt-out: does not affect Google Search or AI Overviews |
| CCBot | Common Crawl | Feeds the open training corpus many models learn from: blocking is a policy choice that slowly reduces what future models know about you |
The two that surprise people
Googlebot is an AI crawler now. AI Overviews and AI Mode can only cite what Googlebot can fetch, so a disallow you added years ago for a staging path may be costing you answers today.
Bingbot matters even if you have never thought about Bing. Copilot answers straight out of Bing’s index, and Bing still feeds part of what ChatGPT retrieves, alongside OpenAI’s own OAI-SearchBot. Blocked there costs you answers in a place you were not watching.
Where robots.txt files usually go wrong
Open yourdomain.com/robots.txt and read it with these four questions in mind. Most sites fail on the second.
- 1
Is there a blanket disallow?
A User-agent: * group with a Disallow: / line blocks everything that has no more specific group of its own. Some AI bots have their own group; several do not.
- 2
Which group actually applies?
Crawlers obey the single most specific matching group, and ignore the rest. A permissive * group does not soften a restrictive PerplexityBot group, and vice versa. This is where audits go wrong.
- 3
Are the important pages inside a disallowed path?
A blocked /blog/ or /guides/ prefix takes exactly the pages you want cited out of scope, however good they are.
- 4
Did you block an answer bot thinking it was a training bot?
The classic: OAI-SearchBot disallowed alongside GPTBot. One of those was the intention. The other removed you from ChatGPT’s citations.
# Training crawlers: a deliberate opt-out User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / # Answer crawlers: explicitly welcome User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / # Everything else, including Googlebot and Bingbot User-agent: * Disallow: /admin/ Disallow: /cart/ Allow: / Sitemap: https://example.com/sitemap.xml
Adapt it. Do not paste it. If you are happy to be used for training, drop the first two groups entirely; there is no visibility penalty either way.
A clean robots.txt does not mean you are reachable
robots.txt is a request, and it is only one of several places access gets lost. Every one of these has silently removed a site from AI answers while its robots.txt looked perfect.
CDN and WAF bot rules
Cloudflare, Fastly and friends ship AI-bot blocking as a toggle, and it is sometimes on by default. This overrides anything robots.txt says, because the request never arrives.
Aggressive rate limiting
A crawler that gets 429s often enough backs off and stops returning. No block, no error page, just a slow fade from the index.
Client-side-only rendering
If the content only exists after JavaScript runs, assume some engines will not see it. Server-render anything you want cited.
Login walls and interstitials
Cookie banners, age gates and “subscribe to continue” overlays can leave a crawler with a page containing no content.
noindex and canonical mistakes
A stray noindex on a template, or a canonical pointing at a competitor’s syndicated copy, removes the page as a citable source.
Geographic blocking
Crawlers run from a handful of regions. Blocking traffic outside your market can block the crawler along with it.
The definitive test is your server logs. Filter the last thirty days by user-agent for the tokens in the first table. If a bot never appears, it is not reading your site, and no amount of robots.txt tidying will change that until you find what is stopping it.
Decide once, write it down, review it twice a year
There is no universally correct answer here. There is a correct process: make the training decision deliberately, keep the answer bots open unless you have a real reason not to, and revisit the file as new crawlers appear. They will. An unmaintained robots.txt slowly turns into a block list for engines that did not exist when you wrote it.
Most businesses
Allow everything. You want to be found, cited and recommended; the training question is largely academic for a company whose content is marketing.
Publishers and IP-heavy sites
Block training, allow answering. You keep the citations and the referral traffic while opting out of unpaid training use.
Blocking everything
A defensible choice, but make it knowingly. It means accepting that AI answers about your category will be written entirely by your competitors.
Ten minutes here can be worth a quarter of content.
Keep reading
Sources and further reading
- Google Search Central: Overview of Google crawlers and user agents
- OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User explained
- Perplexity: PerplexityBot and crawler documentation
- RFC 9309: The Robots Exclusion Protocol
Askwords measures a selected set of buyer questions against fresh answers from each engine. AI answers vary by model, time, context and personalization, so a check is evidence of the answers observed, not a universal judgement about a brand.