Agent purpose: definition and formula

We recognize 26 AI and search bots: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Googlebot-Image, Googlebot-Video, Googlebot-News, Storebot-Google, GoogleOther, GoogleOther-Image, GoogleOther-Video, Google-Agent, bingbot, Applebot, Amazonbot, Meta-ExternalAgent, Meta-ExternalFetcher, Bytespider, CCBot, DuckAssistBot, MistralAI-User.

training · AI search index · fetching for a user · classic search engine

The purpose comes from each provider’s own documentation. Identities are read from the User-Agent header and stay claims unless verified.

• Training: GPTBot, ClaudeBot, GoogleOther, GoogleOther-Image, GoogleOther-Video, Amazonbot, Meta-ExternalAgent, Bytespider, CCBot
• AI search index: OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot
• Fetching for a user: ChatGPT-User, Claude-User, Perplexity-User, Google-Agent, Meta-ExternalFetcher, MistralAI-User
• Classic search engine: Googlebot, Googlebot-Image, Googlebot-Video, Googlebot-News, Storebot-Google, bingbot, Applebot

Purpose says what a visit can lead to. Training bots collect pages for model training. AI search index bots build the index an assistant searches when it answers. User fetchers open a page at the moment a person asks an assistant about it. Classic search engines index for regular search.

In our traffic report, AI requests are training, AI search and user fetches; classic search engine bots are listed but left out of the AI totals and of Crawl-to-refer.

Where a provider’s documentation gives no exact match, we use the nearest class: GoogleOther, used for research and one-off crawls, counts as training. Bytespider has no official documentation and is listed as training.

• Who really sent the request: identities are read from the User-Agent header and stay claims (Claimed identity). Where a provider publishes its IP ranges, the report can check addresses against them; some providers publish none. We don’t do reverse DNS or signature checks.
• Bots outside our table: they aren’t counted as AI, and requests with generic automation signs aren’t counted as human page views either (Human page view).
• What the data is finally used for: the purpose is the provider’s stated purpose.

• robots.txt rules per bot name let you treat purposes differently, for example allowing AI search and user fetchers while refusing training. Test a rule with the robots.txt tester and look up a bot with the bot identifier.
• Blocking the agents named on the Search and user agents allowed (robots.txt) page is a critical blocker in the free test.

See also: Identify a bot from its user agent

Agent traffic (from your server logs): What we count when you load an access log, in the project panel or in the free crawl-to-refer tool.

• Claimed identity
• Human page view
• AI-referred visit
• Crawl-to-refer
• Training-only page

Run the free test →All terms in the glossary →