--- title: "Free tools for AI agent readiness · Specoria" description: "Three browser-only tools: test robots.txt against AI agents, identify an AI bot from its user agent, and get crawl-to-refer from your log. No sign-up." source_url: "https://specoria.com/tools/" lang: "en" --- FREE TOOLS Runs in your browser · nothing you paste is sent to us # Free tools for AI agent readiness Three small tools that run entirely in your browser: test a robots.txt against AI agents, identify an AI bot from its user agent, and work out crawl-to-refer from your own server log. No sign-up, no email. ### Your data stays on your device The tools are two script files served from specoria.com (tools.js and tools-core.js). They make no network requests: what you paste or load is read by your browser, processed there and gone when you close the tab. The logic is the same code our free test and project panel use, bundled for the browser. - [robots.txt AI tester](https://specoria.com/tools/#robots-txt-tester) - [AI bot identifier](https://specoria.com/tools/#ai-bot-identifier) - [Crawl-to-refer calculator](https://specoria.com/tools/#crawl-to-refer) - [AI and search bots we recognize](https://specoria.com/tools/#ai-bots) - [AI revenue-at-risk calculator](https://specoria.com/tools/ai-revenue-calculator/) ## AI revenue-at-risk calculator Estimate how much of your monthly revenue arrives through AI assistants and shopping agents: the channel a store agents can’t read misses. It runs in your browser; nothing you type is sent to us. [Open the calculator on its own page →](https://specoria.com/tools/ai-revenue-calculator/) ## robots.txt AI tester Paste a robots.txt and see, bot by bot, whether AI search bots, agents fetching for a user and training crawlers may read your site. Same parser and rules as our free readiness test: the longest matching rule wins, and a bot with its own group ignores the * group (RFC 9309). [How the free test scores this →](https://specoria.com/method/checks/bots-search/) ## AI bot identifier Paste a User-Agent string from your logs. We match it against the table of AI and search bots below: which company, which bot and what it is for. ## Crawl-to-refer calculator How many requests does each AI company’s bots make for every visitor its assistant sends you? Load an access log: it is parsed here, in your browser, with the same code as our project panel’s agent traffic report. `Crawl-to-refer = AI bot requests of a company (training, AI search and user fetches; classic search engine bots excluded) ÷ human page views referred by that company’s assistants (by Referer, or utm_source when the Referer says nothing).` [Definitions in the glossary →](https://specoria.com/glossary/#crawl-to-refer) Read the ratio as a direction, not an exact number: many apps and privacy settings drop the Referer, user agents are claimed rather than verified, and a log covers only the period it spans. ## AI and search bots we recognize The tools, our free test and our traffic report use this one table. Each row comes from the provider’s own documentation. | User-agent token | Company | Purpose | | --- | --- | --- | | `GPTBot` | OpenAI | Model training | | `OAI-SearchBot` | OpenAI | AI search index | | `ChatGPT-User` | OpenAI | Fetching for a user | | `ClaudeBot` | Anthropic | Model training | | `Claude-SearchBot` | Anthropic | AI search index | | `Claude-User` | Anthropic | Fetching for a user | | `PerplexityBot` | Perplexity | AI search index | | `Perplexity-User` | Perplexity | Fetching for a user | | `Googlebot` | Google | Classic search engine | | `Googlebot-Image` | Google | Classic search engine | | `Googlebot-Video` | Google | Classic search engine | | `Googlebot-News` | Google | Classic search engine | | `Storebot-Google` | Google | Classic search engine | | `GoogleOther` | Google | Model training | | `GoogleOther-Image` | Google | Model training | | `GoogleOther-Video` | Google | Model training | | `Google-Agent` | Google | Fetching for a user | | `bingbot` | Microsoft | Classic search engine | | `Applebot` | Apple | Classic search engine | | `Amazonbot` | Amazon | Model training | | `Meta-ExternalAgent` | Meta | Model training | | `Meta-ExternalFetcher` | Meta | Fetching for a user | | `Bytespider` | ByteDance | Model training | | `CCBot` | Common Crawl | Model training | | `DuckAssistBot` | Other AI | AI search index | | `MistralAI-User` | Other AI | Fetching for a user | Google-Extended and Applebot-Extended are robots.txt tokens only: they never send requests and never appear in logs. Their bots arrive as Googlebot and Applebot. Sources: [platform.openai.com](https://platform.openai.com/docs/bots) · [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) · [docs.perplexity.ai](https://docs.perplexity.ai/guides/bots) · [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) · [bing.com](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) · [support.apple.com](https://support.apple.com/en-us/119829) · [developer.amazon.com](https://developer.amazon.com/amazonbot) · [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) · [commoncrawl.org](https://commoncrawl.org/ccbot) · [duckduckgo.com](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) · [docs.mistral.ai](https://docs.mistral.ai/robots) [Run the free readiness test →](https://specoria.com/#contact)[Watch agents shop your store →](https://specoria.com/agent-simulator/)