How the engines work
Should I block AI crawlers like GPTBot?
Not if you want to be cited. The crawlers are separate: GPTBot collects training data, OAI-SearchBot powers ChatGPT search, and ChatGPT-User fetches pages on request; Google-Extended only governs Gemini training and has no effect on Search or AI Overviews, which use Googlebot. Block training bots if you must; never block the search ones.
Updated · Citemark
Who crawls what
Each company publishes its user agents and what they are for. The ones that matter for being cited are the search and on-demand fetchers; the training crawlers only affect whether your text helps train future models.
- OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search index), ChatGPT-User (fetch on request).
- Google: Googlebot (Search, AI Overviews, AI Mode), Google-Extended (a control token for Gemini training only).
- Perplexity: PerplexityBot (index) and Perplexity-User (fetch on request).
- Anthropic: ClaudeBot (training), Claude-SearchBot and Claude-User (search and on-request).
- Microsoft: Bingbot — which also feeds Copilot and, through Bing, ChatGPT search.
The blocks you didn’t know you had
robots.txt isn’t the only gate. Hosting and CDN dashboards — Vercel, Cloudflare and others — offer one-click “block AI bots” switches that return errors to every AI crawler regardless of robots.txt. Check those before assuming you’re open. A retrievability audit compares what each crawler actually receives with what a user sees.
Sources
See what AI says about you right now
The free audit runs the prompts for your category across every engine and shows, screenshot by screenshot, who gets cited — and the three things to fix first.