Search and user agents allowed (robots.txt)
Why it matters If OAI-SearchBot is blocked the store doesn’t appear in ChatGPT search; if Googlebot is blocked Google AI Mode and Gemini can’t read the product page.
- Category
- Access
- Weight
- 8 of 100 points
- Critical blocker
- Yes. If it fails, the score is capped at 59; with two or more blockers, at 39.
- Applies to
- Online stores · Hotels and lodging · Service businesses
- Effort
- 15 minutes · developer or SEO
- Score model
- v2.4
How we measure it
We read robots.txt and test the root and the product page’s path for search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot) and user agents (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent). Blocking any of them fails the check; blocking OAI-SearchBot or Googlebot is a critical blocker. No robots.txt passes; an unreadable one (server error, timeout) is not verifiable. Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) and other AI bots (Applebot-Extended, Meta-ExternalAgent, Amazonbot, MistralAI-User, Bytespider) don’t count and are shown as information rows; so are agent-specific groups that override the general (*) rules and any Content-Signal declaration.
On hotel and service sites this check is measured on the homepage instead of a product page.
What the result page says
These are the sentences in the test result; “…” is replaced by what we found on your site.
- Passedrobots.txt doesn’t block OAI-SearchBot, ChatGPT-User, Claude, Perplexity, Googlebot, Google-Agent, Bingbot or Applebot on the homepage or the product page.
- Failedrobots.txt blocks these agents: …. They can’t read or recommend your store.
How to fix it
Target: Search and user agents are explicitly allowed. You can still block training bots (GPTBot, ClaudeBot, Google-Extended); that doesn’t cut visibility.
Example code
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Training bots are optional:
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlExample values (example.com, product name, price) are placeholders; replace them with your own.
How to verify it yourself
Update robots.txt and run the test again; check Google Search Console’s robots.txt report too.
Fix it with an AI coding agent
Paste this into Claude Code, Cursor or another coding agent working on your store’s code. Review the change before you deploy it.
You are working on my online store. Specoria’s agent readiness test checks the item below. Fix it so AI shopping agents can read the store and act on it.
## 1. Search and user agents allowed (robots.txt) (bots_search)
Target: Search and user agents are explicitly allowed. You can still block training bots (GPTBot, ClaudeBot, Google-Extended); that doesn’t cut visibility.
Example fix (adapt it to this codebase and platform; example values are placeholders):
```
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Training bots are optional:
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
```
Reference: https://specoria.com/method/checks/bots-search/
## Rules
- First look through the codebase and tell me which files or templates you will change, before editing anything.
- Change only what these checks need, and keep the visible page and the structured data saying the same thing.
- Never invent prices, stock, shipping or return terms, reviews or product identifiers (GTIN): take them from the store’s real data or ask me.
- Don’t block or slow down real visitors and don’t add tracking.
- After deploying, run the free test again at https://specoria.com/ to verify the fix.Specifications and guidance
The check is based on these open sources; links go to the publisher’s own pages.
Where it sits in the 196-criteria framework
The result of this check counts as evidence for these items of the readiness framework:
1.1OAI-SearchBot allowed in robots.txt