Free tools for AI agent readiness
Three small tools that run entirely in your browser: test a robots.txt against AI agents, identify an AI bot from its user agent, and work out crawl-to-refer from your own server log. No sign-up, no email.
AI revenue-at-risk calculator
Estimate how much of your monthly revenue arrives through AI assistants and shopping agents: the channel a store agents can’t read misses. It runs in your browser; nothing you type is sent to us.
robots.txt AI tester
Paste a robots.txt and see, bot by bot, whether AI search bots, agents fetching for a user and training crawlers may read your site. Same parser and rules as our free readiness test: the longest matching rule wins, and a bot with its own group ignores the * group (RFC 9309).
AI bot identifier
Paste a User-Agent string from your logs. We match it against the table of AI and search bots below: which company, which bot and what it is for.
Crawl-to-refer calculator
How many requests does each AI company’s bots make for every visitor its assistant sends you? Load an access log: it is parsed here, in your browser, with the same code as our project panel’s agent traffic report.
Crawl-to-refer = AI bot requests of a company (training, AI search and user fetches; classic search engine bots excluded) ÷ human page views referred by that company’s assistants (by Referer, or utm_source when the Referer says nothing).
Read the ratio as a direction, not an exact number: many apps and privacy settings drop the Referer, user agents are claimed rather than verified, and a log covers only the period it spans.
AI and search bots we recognize
The tools, our free test and our traffic report use this one table. Each row comes from the provider’s own documentation.
| User-agent token | Company | Purpose |
|---|---|---|
GPTBot | OpenAI | Model training |
OAI-SearchBot | OpenAI | AI search index |
ChatGPT-User | OpenAI | Fetching for a user |
ClaudeBot | Anthropic | Model training |
Claude-SearchBot | Anthropic | AI search index |
Claude-User | Anthropic | Fetching for a user |
PerplexityBot | Perplexity | AI search index |
Perplexity-User | Perplexity | Fetching for a user |
Googlebot | Classic search engine | |
Googlebot-Image | Classic search engine | |
Googlebot-Video | Classic search engine | |
Googlebot-News | Classic search engine | |
Storebot-Google | Classic search engine | |
GoogleOther | Model training | |
GoogleOther-Image | Model training | |
GoogleOther-Video | Model training | |
Google-Agent | Fetching for a user | |
bingbot | Microsoft | Classic search engine |
Applebot | Apple | Classic search engine |
Amazonbot | Amazon | Model training |
Meta-ExternalAgent | Meta | Model training |
Meta-ExternalFetcher | Meta | Fetching for a user |
Bytespider | ByteDance | Model training |
CCBot | Common Crawl | Model training |
DuckAssistBot | Other AI | AI search index |
MistralAI-User | Other AI | Fetching for a user |
Google-Extended and Applebot-Extended are robots.txt tokens only: they never send requests and never appear in logs. Their bots arrive as Googlebot and Applebot.
Sources: platform.openai.com · support.claude.com · docs.perplexity.ai · developers.google.com · bing.com · support.apple.com · developer.amazon.com · developers.facebook.com · commoncrawl.org · duckduckgo.com · docs.mistral.ai