Should You Allow GPTBot, PerplexityBot and ClaudeBot?
Each AI crawler named, what it actually does, the training versus retrieval distinction that decides the answer, and how to configure robots.txt.
Published 2026-11-01 · 4 min read
Most sites block AI crawlers by default or never consider them, and the decision is usually made without knowing that the crawlers do two completely different things. One group trains models on your content. The other fetches pages at the moment someone asks a question — and that second group is what produces citations and referral traffic.
You can allow one and not the other. This is what each does and how to decide.
The distinction that matters
| Type | When it runs | What you get | What you give |
|---|---|---|---|
| Training crawlers | Periodically, in bulk | Nothing directly | Your content may be used to train models |
| Retrieval crawlers | At answer time, per query | A citation, a referral, visibility in answers | A page fetch |
| User-triggered fetchers | When a user pastes your URL | The user sees your content rendered | A page fetch |
The crawlers, named
- GPTBot — training crawler for OpenAI models. Bulk, periodic, no direct return.
- OAI-SearchBot — retrieval for search-style answers. This one fetches when a user asks something, and it is what makes a ChatGPT answer able to cite you.
- ChatGPT-User — fetches pages a user asks about directly during a conversation.
- PerplexityBot and Perplexity-User — retrieval and user-triggered fetch for Perplexity, whose whole product is cited answers. Blocking these removes you from that surface entirely.
- ClaudeBot and Claude-Web — Anthropic's crawler and web fetch.
- Google-Extended — the control for whether your content is used for Google's Gemini models. Separate from Googlebot, so blocking it does not affect normal search indexing.
- Applebot-Extended — Apple's equivalent control for its AI features.
- Amazonbot, cohere-ai, Bytespider, meta-externalagent, YouBot — the rest of the field, smaller but growing.
The three positions
| Position | Configuration | Who it suits |
|---|---|---|
| Allow everything | No AI-specific rules; default allow | Sites that want maximum citation and referral, and are comfortable with training use |
| Allow retrieval, block training | Permit the retrieval and user-fetch agents, disallow the training ones | Sites that want citations but not training use — the position most publishers should consider |
| Block everything | Disallow all listed agents | Sites with proprietary data, licensing constraints, or content whose value depends on not being summarised |
The case for allowing retrieval
- It is the only channel where a new domain can win on quality rather than authority. Search ranking depends heavily on domain authority; retrieval and synthesis depend much more on whether the page is reachable and clearly written.
- The traffic is high-intent. Someone asking an assistant for a tool in your category is further along than someone browsing a search results page.
- Blocking it is invisible to you. You do not get a notification that you were excluded from an answer. Most sites that block retrieval crawlers have no idea what it costs them.
- Your competitors are mostly not doing it. Which is the whole argument.
The case for caution
- Training use is a real transfer of value. Your content may end up improving a model that competes with you, with no compensation and no attribution. Blocking training crawlers is a legitimate position.
- Answer engines reduce clicks. If your business depends on ad revenue from page views, a cited answer may satisfy the query without a visit. For a product site this is much less of a concern than for a publisher.
- Content that is expensive to produce deserves a deliberate choice. Original research, proprietary data and deep editorial justify more care than a marketing page.
- Server load. Retrieval crawlers fetch on demand and can be bursty. Usually trivial, occasionally not.
Configuration
The mechanism is robots.txt, with explicit rules per user agent. The shape:
- Start from a default allow for everything except machine paths — API endpoints, checkout, account pages. Blocking by default and allowing selectively is how sites accidentally hide their public content.
- Add explicit allow rules for the retrieval agents you want. Explicit is better than relying on the default, because it documents intent and survives a later change to the default.
- Add explicit disallow for training agents if you are taking the middle position.
- Keep your sitemap referenced in the same file.
- Never block a page you have marked noindex. A blocked page cannot be read, so the noindex tag is never seen and the page may be indexed anyway from external links — the opposite of what you intended.
How to check what is happening
- Read your server logs by user agent. The only direct evidence of which crawlers are reaching you, and how often.
- Test your robots.txt with the standard validation tools before deploying a change.
- Ask the assistants whether they can see a specific page. Crude, but a useful sanity check after a change.
- Audit quarterly. Crawler names change and new ones appear. A robots file written once and never revisited drifts.
The question is not whether to allow AI crawlers. It is whether you want to appear in answers, and whether you are willing to let your content train the models that produce them — two separate decisions that most sites accidentally make together.
More in SEO & GEO
- Entity Consistency: Why AI Models Fail to Attribute Your ProductWhen a model cannot tell your mentions describe one product, it attributes none of them. The signals that resolve an entity, and how to audit yours.
- How to Measure AI Referral Traffic When UTMs Are StrippedAI assistants strip UTM parameters, so referrer segmentation is the only reliable method. The setup, the domains to watch, and how to read the numbers.
- The 10-Page SEO Starter Kit for a New AI ProductThe ten pages a new AI product should have before writing anything else, what each one is for, and the order to build them in.