The AI crawler list: every user agent, one page

Block one of these and you're out of that engine's answers

These are the crawlers that build the index an answer engine picks its sources from. If you want ChatGPT or Google to recommend you, they have to be allowed in. There's no way around that.

These show up because a real person asked about you

Strictly speaking these aren't crawlers. Someone asks ChatGPT or Claude a question, the answer needs your page, and the fetcher goes and reads it right then. OpenAI, Perplexity and Meta all say plainly that robots.txt may not apply to these, so blocking them is mostly a gesture. What actually helps is a page that loads fast and has the content in the HTML.

Blocking these is a policy decision, and it won't cost you a single answer

Disallow any of these and your pages stay out of future model training. That's all it does. You keep every citation. Two of the tokens (Google-Extended and Applebot-Extended) aren't even crawlers, they're switches that Googlebot and Applebot read to decide what they may do with pages they already fetched.

Two files that cover most situations

Both blocks are generated straight from the tables above, so when we add a crawler it lands in the snippet too. Every token gets its own group. That's deliberate: a crawler follows the most specific group that names it, so an explicit line beats hoping the * fallback covers it, and anyone auditing your file sees a decision rather than a shrug.

# Answer engines and browsing: allowed
User-agent: OAI-SearchBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
User-agent: MistralAI-Index
User-agent: DuckAssistBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: MistralAI-User
Allow: /

# Training crawlers and training controls: disallowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

This is the one most publishers want. You keep every citation and every referral, and nobody trains on your pages. Your normal search rules stay in the User-agent: * group you already have.

# Every documented AI crawler, explicitly allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: Googlebot
User-agent: Google-Extended
User-agent: Bingbot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: MistralAI-Training
User-agent: DuckAssistBot
User-agent: CCBot
Allow: /

If your website is your marketing, the training question is mostly academic and this is the simpler file. Technically it says nothing an empty file wouldn't. But it says it on purpose, and when someone (or some tool) audits your site, it reads as a decision you made.

Anyone can type a user-agent string

Scrapers call themselves Googlebot or GPTBot because those names usually get waved through. So before you decide from your logs that an answer engine is reading your site, or that one is hammering it, check the request's IP against the owner's published list (the "IP list" links in the tables). OpenAI, Anthropic, Perplexity, Google, Microsoft and Common Crawl all publish one. If a request claims a token but comes from outside that list, it isn't that crawler, and you can block it at the firewall without touching robots.txt.

The opposite check matters more for visibility. Filter the last thirty days of logs for the answer engine tokens in the first table. If one never shows up, it isn't reading you, and the reason is somewhere before robots.txt: a firewall rule, a "block AI bots" switch on your CDN, or a page that only exists once JavaScript has run.

What changed, and when we last checked

• 18 September 2026. First edition. We checked every row against the owner's own docs. One thing that surprised us: Anthropic now describes ClaudeBot as its training crawler, with Claude-SearchBot and Claude-User doing the answering. Older lists (ours included, until today) had that backwards.

Vendors add and rename crawlers a few times a year, so this page will drift if we don't keep at it. If you spot a missing token or a docs link that has moved, tell us and we'll add it, with credit.