okaihub

Should You Allow GPTBot, PerplexityBot and ClaudeBot?

Each AI crawler named, what it actually does, the training versus retrieval distinction that decides the answer, and how to configure robots.txt.

Published 2026-11-01 · 4 min read

Most sites block AI crawlers by default or never consider them, and the decision is usually made without knowing that the crawlers do two completely different things. One group trains models on your content. The other fetches pages at the moment someone asks a question — and that second group is what produces citations and referral traffic.

You can allow one and not the other. This is what each does and how to decide.

The distinction that matters

TypeWhen it runsWhat you getWhat you give
Training crawlersPeriodically, in bulkNothing directlyYour content may be used to train models
Retrieval crawlersAt answer time, per queryA citation, a referral, visibility in answersA page fetch
User-triggered fetchersWhen a user pastes your URLThe user sees your content renderedA page fetch
The middle row is the one with a return. Blocking it to avoid training exposure also removes your chance of being cited.

The crawlers, named

  • GPTBot — training crawler for OpenAI models. Bulk, periodic, no direct return.
  • OAI-SearchBot — retrieval for search-style answers. This one fetches when a user asks something, and it is what makes a ChatGPT answer able to cite you.
  • ChatGPT-User — fetches pages a user asks about directly during a conversation.
  • PerplexityBot and Perplexity-User — retrieval and user-triggered fetch for Perplexity, whose whole product is cited answers. Blocking these removes you from that surface entirely.
  • ClaudeBot and Claude-Web — Anthropic's crawler and web fetch.
  • Google-Extended — the control for whether your content is used for Google's Gemini models. Separate from Googlebot, so blocking it does not affect normal search indexing.
  • Applebot-Extended — Apple's equivalent control for its AI features.
  • Amazonbot, cohere-ai, Bytespider, meta-externalagent, YouBot — the rest of the field, smaller but growing.

The three positions

PositionConfigurationWho it suits
Allow everythingNo AI-specific rules; default allowSites that want maximum citation and referral, and are comfortable with training use
Allow retrieval, block trainingPermit the retrieval and user-fetch agents, disallow the training onesSites that want citations but not training use — the position most publishers should consider
Block everythingDisallow all listed agentsSites with proprietary data, licensing constraints, or content whose value depends on not being summarised
The middle option is the one most often missed, because it requires knowing which crawler is which.

The case for allowing retrieval

  • It is the only channel where a new domain can win on quality rather than authority. Search ranking depends heavily on domain authority; retrieval and synthesis depend much more on whether the page is reachable and clearly written.
  • The traffic is high-intent. Someone asking an assistant for a tool in your category is further along than someone browsing a search results page.
  • Blocking it is invisible to you. You do not get a notification that you were excluded from an answer. Most sites that block retrieval crawlers have no idea what it costs them.
  • Your competitors are mostly not doing it. Which is the whole argument.

The case for caution

  • Training use is a real transfer of value. Your content may end up improving a model that competes with you, with no compensation and no attribution. Blocking training crawlers is a legitimate position.
  • Answer engines reduce clicks. If your business depends on ad revenue from page views, a cited answer may satisfy the query without a visit. For a product site this is much less of a concern than for a publisher.
  • Content that is expensive to produce deserves a deliberate choice. Original research, proprietary data and deep editorial justify more care than a marketing page.
  • Server load. Retrieval crawlers fetch on demand and can be bursty. Usually trivial, occasionally not.

Configuration

The mechanism is robots.txt, with explicit rules per user agent. The shape:

  • Start from a default allow for everything except machine paths — API endpoints, checkout, account pages. Blocking by default and allowing selectively is how sites accidentally hide their public content.
  • Add explicit allow rules for the retrieval agents you want. Explicit is better than relying on the default, because it documents intent and survives a later change to the default.
  • Add explicit disallow for training agents if you are taking the middle position.
  • Keep your sitemap referenced in the same file.
  • Never block a page you have marked noindex. A blocked page cannot be read, so the noindex tag is never seen and the page may be indexed anyway from external links — the opposite of what you intended.

How to check what is happening

  • Read your server logs by user agent. The only direct evidence of which crawlers are reaching you, and how often.
  • Test your robots.txt with the standard validation tools before deploying a change.
  • Ask the assistants whether they can see a specific page. Crude, but a useful sanity check after a change.
  • Audit quarterly. Crawler names change and new ones appear. A robots file written once and never revisited drifts.

The question is not whether to allow AI crawlers. It is whether you want to appear in answers, and whether you are willing to let your content train the models that produce them — two separate decisions that most sites accidentally make together.

More in SEO & GEO