--- title: "Search and user agents allowed (robots.txt) · Specoria" description: "If OAI-SearchBot is blocked the store doesn’t appear in ChatGPT search; if Googlebot is blocked Google AI Mode and Gemini can’t read the product page." source_url: "https://specoria.com/method/checks/bots-search/" lang: "en" --- [← All checks](https://specoria.com/method/checks/) bots_search # Search and user agents allowed (robots.txt) Why it matters If OAI-SearchBot is blocked the store doesn’t appear in ChatGPT search; if Googlebot is blocked Google AI Mode and Gemini can’t read the product page. **Category** Access **Weight** 8 of 100 points **Critical blocker** Yes. If it fails, the score is capped at 59; with two or more blockers, at 39. **Applies to** Online stores · Hotels and lodging · Service businesses **Effort** 15 minutes · developer or SEO **Score model** v2.4 ## How we measure it We read robots.txt and test the root and the product page’s path for search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot) and user agents (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent). Blocking any of them fails the check; blocking OAI-SearchBot or Googlebot is a critical blocker. No robots.txt passes; an unreadable one (server error, timeout) is not verifiable. Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) and other AI bots (Applebot-Extended, Meta-ExternalAgent, Amazonbot, MistralAI-User, Bytespider) don’t count and are shown as information rows; so are agent-specific groups that override the general (*) rules and any Content-Signal declaration. On hotel and service sites this check is measured on the homepage instead of a product page. ### What the result page says These are the sentences in the test result; “…” is replaced by what we found on your site. - Passed robots.txt doesn’t block OAI-SearchBot, ChatGPT-User, Claude, Perplexity, Googlebot, Google-Agent, Bingbot or Applebot on the homepage or the product page. - Failed robots.txt blocks these agents: …. They can’t read or recommend your store. ## How to fix it **Target:** Search and user agents are explicitly allowed. You can still block training bots (GPTBot, ClaudeBot, Google-Extended); that doesn’t cut visibility. ### Example code `User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / # Training bots are optional: User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml` Example values (example.com, product name, price) are placeholders; replace them with your own. ### How to verify it yourself Update robots.txt and run the test again; check Google Search Console’s robots.txt report too. ### Fix it with an AI coding agent Paste this into Claude Code, Cursor or another coding agent working on your store’s code. Review the change before you deploy it. `You are working on my online store. Specoria’s agent readiness test checks the item below. Fix it so AI shopping agents can read the store and act on it. ## 1. Search and user agents allowed (robots.txt) (bots_search) Target: Search and user agents are explicitly allowed. You can still block training bots (GPTBot, ClaudeBot, Google-Extended); that doesn’t cut visibility. Example fix (adapt it to this codebase and platform; example values are placeholders): ``` User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / # Training bots are optional: User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml ``` Reference: https://specoria.com/method/checks/bots-search/ ## Rules - First look through the codebase and tell me which files or templates you will change, before editing anything. - Change only what these checks need, and keep the visible page and the structured data saying the same thing. - Never invent prices, stock, shipping or return terms, reviews or product identifiers (GTIN): take them from the store’s real data or ask me. - Don’t block or slow down real visitors and don’t add tracking. - After deploying, run the free test again at https://specoria.com/ to verify the fix.` ## Specifications and guidance - [RFC 9309: Robots Exclusion Protocol ↗](https://www.rfc-editor.org/rfc/rfc9309) - [OpenAI: crawlers and user agents ↗](https://platform.openai.com/docs/bots) The check is based on these open sources; links go to the publisher’s own pages. ## Where it sits in the 196-criteria framework The result of this check counts as evidence for these items of the readiness framework: - `1.1` OAI-SearchBot allowed in robots.txt [The framework and its source →](https://specoria.com/method/#criteria) [Next →Is bot protection open to agents?](https://specoria.com/method/checks/waf-access/) [Run the free test](https://specoria.com/method/checks/bots-search/#test) [See the sample report →](https://specoria.com/reports/sample-report/) ## Related - [Is bot protection open to agents?](https://specoria.com/method/checks/waf-access/) - [Product name and price readable without JavaScript](https://specoria.com/method/checks/no-js/) - [Agent protocol map: UCP, ACP, AP2, MCP, WebMCP](https://specoria.com/agent-protocols/) What each protocol does and what the free test checks - [Scan API, MCP server and GitHub Action](https://specoria.com/developers/) Run the free test from your own scripts or CI FREE TEST ## Test your store now Enter your store’s address: 27 checks on your public pages, the way a shopping agent reads them. The free test runs in your browser and needs JavaScript. [Request the free deep audit by email instead →](https://specoria.com/#contact) Your result opens on its own page. Want the full picture? The free deep audit and panel come next, by email.