Every AI crawler that matters in 2026
If AI crawlers can't fetch your pages, you cannot be cited — no matter how good your content is. This is the single most common GEO failure we find in audits, usually caused by a CDN "block AI bots" preset someone enabled and forgot.
The crawlers, by company
Three roles to understand: training bots collect data for future models, search/index bots build retrieval indexes, and user-fetch agents grab a page live because a user's question needs it. For GEO you generally want all three allowed.
| User agent | Company | Role | Why it matters for GEO |
|---|---|---|---|
GPTBot | OpenAI | Training | Feeds future GPT models' memory of your brand |
OAI-SearchBot | OpenAI | Search index | Powers ChatGPT search results and link citations |
ChatGPT-User | OpenAI | User fetch | Live page visits when ChatGPT browses for an answer |
ClaudeBot | Anthropic | Training | Claude's model knowledge of your site |
Claude-User | Anthropic | User fetch | Live fetches during Claude web search |
anthropic-ai | Anthropic | Legacy/training | Older UA still seen in logs; allow for safety |
PerplexityBot | Perplexity | Search index | Perplexity's own index — its answers cite from it heavily |
Perplexity-User | Perplexity | User fetch | Live retrieval during a Perplexity answer |
Google-Extended | Training (Gemini) | Controls Gemini training use; doesn't affect Google Search rank | |
GoogleOther | R&D fetch | Misc Google product fetches, incl. AI experiments | |
Googlebot | Search index | AI Overviews are built on the normal search index — regular SEO access is your GEO access | |
Bingbot | Microsoft | Search index | Copilot answers ride on the Bing index |
Applebot-Extended | Apple | Training | Apple Intelligence training opt-in |
Amazonbot | Amazon | Training/index | Alexa and Amazon AI answers |
meta-externalagent | Meta | Training/index | Llama models and Meta AI answers |
cohere-ai | Cohere | Training | Enterprise RAG deployments built on Cohere |
Bytespider | ByteDance | Training | Doubao/CN-market AI; relevant if China matters to you |
DuckAssistBot | DuckDuckGo | Search/answers | DuckAssist AI answers |
robots.txt: welcome them all
Copy-paste block (also available with one click in our free tool):
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: meta-externalagent
Allow: /Three gotchas that silently block you
- CDN bot protection. Cloudflare, Akamai and friends ship one-toggle "block AI scrapers" features. robots.txt says Allow, the firewall says 403, and you never notice. Check your WAF rules, not just robots.txt.
- JavaScript-only content. Most AI crawlers don't execute JS. If your product page is an empty div until React hydrates, AI sees the empty div. Server-render anything you want quoted.
- Verify with logs, not vibes. grep your access logs for the user agents above. Seeing
GPTBotfetch your pricing page is the ground truth that AI can read you. UpGeo's crawler tracking automates exactly this.
Should you ever block them?
Publishers monetizing content may reasonably block training bots while allowing search/user-fetch ones (that's what Google-Extended exists for — blocking it doesn't touch your search ranking). But if your goal is being recommended when buyers ask AI what to use, blocking is self-sabotage: you're removing yourself from the answer.
UpGeo's audit checks crawler access, JS rendering and server logs — then tracks AI bot visits continuously on the Growth plan.
Check my AI access