---
title: "Crawl-to-refer: definition and formula · Specoria"
description: "How many requests an AI company’s bots make for every human visit its assistants send you. Classic search bots are left out; works from your own log."
source_url: "https://specoria.com/glossary/crawl-to-refer/"
lang: "en"
---

GLOSSARY Score model v2.6 · last reviewed October 8, 2026

# Crawl-to-refer: definition and formula

How many requests an AI company’s bots make for every human visit its assistants send you.

## Definition

Classic search engine bots are left out. With no referred visit, the ratio isn’t calculated.

## Formula

`crawl-to-refer = AI bot requests of a company ÷ AI-referred visits from that company’s assistants`

## Example

OpenAI bots made 1,200 requests (training, AI search and user fetches; Googlebot’s requests don’t count) and ChatGPT sent 40 visits: 1,200 ÷ 40 = 30 requests per visit.

## How to read it

Higher means more bot requests for each visit sent back. It is computed per company, for example OpenAI’s training, search and user bots against visits from ChatGPT. When a company sent no visits, there is no ratio, only a request count.

Assistants are matched to companies: ChatGPT to OpenAI, Claude to Anthropic, Perplexity to Perplexity, Gemini to Google, Copilot to Microsoft and Meta AI to Meta; the rest are grouped as other AI.

Microsoft’s only bot in our table, bingbot, is a classic search engine, so Copilot visits are set against zero AI bot requests. We set no good or bad range: compare a company with itself over time and with the other companies in the same log.

## What it does not tell you

- The exact ratio: AI-referred visits are a lower bound, so missing visits shrink the denominator and the ratio tends to be overstated.
- Whether the bots are real: requests are counted by claimed identity ([Claimed identity](https://specoria.com/glossary/#claimed-identity)), including any that fail our IP check.
- Which requests can lead to a visit: training requests and user fetches count the same; the [Agent purpose](https://specoria.com/glossary/agent-purpose/) breakdown shows the mix.
- Anything outside your log: it covers only the period the log covers.

## What moves it

- Fewer requests from a company’s bots lower it, for example robots.txt rules per [Agent purpose](https://specoria.com/glossary/agent-purpose/); more visits from its assistant lower it too ([AI-referred visit](https://specoria.com/glossary/ai-referred-visit/)).
- Calculate it from your own access log with the [crawl-to-refer tool](https://specoria.com/tools/#crawl-to-refer); the file is read in your browser.

See also: [Calculate it from your own log, in your browser](https://specoria.com/tools/)

## Related terms

Agent traffic (from your server logs): What we count when you load an access log, in the project panel or in the free crawl-to-refer tool.

- [Agent purpose](https://specoria.com/glossary/agent-purpose/)
- [Claimed identity](https://specoria.com/glossary/#claimed-identity)
- [Human page view](https://specoria.com/glossary/#human-page-view)
- [AI-referred visit](https://specoria.com/glossary/ai-referred-visit/)
- [Training-only page](https://specoria.com/glossary/#training-only-page)

[Run the free test →](https://specoria.com/#contact)[All terms in the glossary →](https://specoria.com/glossary/)