Specoria
Menu
TR Free test
← All checks
waf_access

Is bot protection open to agents?

Why it matters An agent that sees a 403 or a challenge page can’t compare the store; browser agents (ChatGPT agent, Comet) get stuck too.

Category
Access
Weight
8 of 100 points
Critical blocker
Yes. If it fails, the score is capped at 59; with two or more blockers, at 39.
Applies to
Online stores · Hotels and lodging · Service businesses
Effort
1-2 hours · infrastructure or CDN admin
Score model
v2.4

How we measure it

We request the homepage and the product page with ordinary browser headers. A hard refusal (e.g. HTTP 403) fails the check and is a critical blocker. A challenge page or CAPTCHA is a “possible block”: a partial pass, not a critical blocker, since we can’t see from outside what verified agents get. A rate limit (HTTP 429) isn’t counted as a block: we retry after the Retry-After wait; if the page opens it passes, if it’s still limited the check is not verifiable and doesn’t count. If the homepage is blocked from Cloudflare’s network but opens from a second network, it’s a partial pass. If only the one request that identifies itself as SpecoriaBot is refused, that’s also a “possible block” (partial pass): we can’t send requests from verified bot IPs.

On hotel and service sites this check is measured on the homepage instead of a product page.

What the result page says

These are the sentences in the test result; “…” is replaced by what we found on your site.

  • PassedThe homepage and product page responded normally to an automated request; we didn’t hit a challenge wall.
  • PassedThe first request hit a rate limit (……); after a short wait the page opened normally. A rate limit isn’t a block: answering HTTP 429 with a Retry-After header is the right behaviour for agents.
  • PartialOur scan from Cloudflare’s network was refused (……), but … opened normally from a second data center and the scan was completed from there. The block is specific to Cloudflare traffic; most agents come from other networks, though browser agents running on Cloudflare’s infrastructure may get stuck.
  • PartialThe page opened with a browser identity, but a request that identified itself as a bot (SpecoriaBot) was refused (……). This is a possible block: a rule that blocks bots by identity can also stop AI agents such as OAI-SearchBot; real agents come from verified IPs and may be treated differently.
  • Partial… showed our scan a challenge page (……). This is a possible block: agent crawlers that don’t run JavaScript can’t get past it. If verified AI bots are allowed they may see the normal page; we can’t verify that from outside.
  • Partial… showed our scan a challenge page both from Cloudflare’s network and from a second data center (……). This is a possible block: agent crawlers that don’t run JavaScript can’t get past it. If verified AI bots are allowed they may see the normal page; we can’t verify that from outside.
  • Partial… showed our scan a challenge page (……). This is a possible block: an agent may leave without seeing the product. If verified AI bots are allowed they may see the page; we can’t verify that from outside.
  • Failed… turned our scan away (……). The site’s security layer stops automated requests, and shopping agents may hit the same wall.
  • Failed… turned our scan away (……). An agent leaves without seeing the product.
  • Failed… turned our scan away both from Cloudflare’s network and from a second data center (……). The site’s security layer blocks data-center traffic in general, so shopping agents will most likely hit the same wall.

How to fix it

Target: Bot protection stops only malicious traffic; verified search and user agents get the normal page (HTTP 200). Risk-based checks and rate limits (HTTP 429 + Retry-After) replace a CAPTCHA at every step.

Example code

General

Subject: Allow verified AI agents (example.com)

An agent-readiness test showed that our bot protection stops automated requests.
So that search and shopping agents can read our product pages, let's allow these bots:
- OAI-SearchBot, ChatGPT-User (OpenAI)
- Claude-SearchBot, Claude-User (Anthropic)
- PerplexityBot, Perplexity-User (Perplexity)
- Googlebot, Bingbot, Applebot

Steps (no bot-protection vendor was detected):
1. In the bot or WAF panel, allow "verified" / known good bots.
2. Add an exception for the search and user agents above; you can keep training bots (GPTBot, ClaudeBot) blocked if you like.
3. On product, cart and policy pages, use rate limits instead of challenge pages.

Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish.
Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs.
(Menu names can differ by the vendor's interface version and plan.)

Cloudflare

Subject: Allow verified AI agents (example.com)

An agent-readiness test showed that our bot protection stops automated requests.
So that search and shopping agents can read our product pages, let's allow these bots:
- OAI-SearchBot, ChatGPT-User (OpenAI)
- Claude-SearchBot, Claude-User (Anthropic)
- PerplexityBot, Perplexity-User (Perplexity)
- Googlebot, Bingbot, Applebot

Steps (a Cloudflare signature was seen):
1. Security → Bots: allow verified bots ("Allow verified bots" in Super Bot Fight Mode).
2. AI Crawl Control: set OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User to "Allow".
3. If needed, a custom WAF rule: (cf.verified_bot_category in {"AI Search" "AI Assistant" "Search Engine Crawler"}) → Skip.

Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish.
Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs.
(Menu names can differ by the vendor's interface version and plan.)

Akamai

Subject: Allow verified AI agents (example.com)

An agent-readiness test showed that our bot protection stops automated requests.
So that search and shopping agents can read our product pages, let's allow these bots:
- OAI-SearchBot, ChatGPT-User (OpenAI)
- Claude-SearchBot, Claude-User (Anthropic)
- PerplexityBot, Perplexity-User (Perplexity)
- Googlebot, Bingbot, Applebot

Steps (an Akamai signature was seen):
1. Bot Manager → Akamai-categorized known bots: set the search engine and AI search/assistant categories to "Allow".
2. Make sure no challenge or tarpit action applies to those categories.
3. You can decide separately on training bots; keep search and user agents open.

Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish.
Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs.
(Menu names can differ by the vendor's interface version and plan.)

DataDome

Subject: Allow verified AI agents (example.com)

An agent-readiness test showed that our bot protection stops automated requests.
So that search and shopping agents can read our product pages, let's allow these bots:
- OAI-SearchBot, ChatGPT-User (OpenAI)
- Claude-SearchBot, Claude-User (Anthropic)
- PerplexityBot, Perplexity-User (Perplexity)
- Googlebot, Bingbot, Applebot

Steps (a DataDome signature was seen):
1. Bot Protection → allowlist (good bots): set OpenAI, Anthropic and Perplexity search and user agents to "Allow".
2. Check that the CAPTCHA rule is off for verified bots.

Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish.
Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs.
(Menu names can differ by the vendor's interface version and plan.)

Imperva

Subject: Allow verified AI agents (example.com)

An agent-readiness test showed that our bot protection stops automated requests.
So that search and shopping agents can read our product pages, let's allow these bots:
- OAI-SearchBot, ChatGPT-User (OpenAI)
- Claude-SearchBot, Claude-User (Anthropic)
- PerplexityBot, Perplexity-User (Perplexity)
- Googlebot, Bingbot, Applebot

Steps (an Imperva signature was seen):
1. Bot Management → client classification: allow search bots and AI agents as good bots.
2. Make sure Security Rules don't apply a challenge (JavaScript/CAPTCHA) to those classes.

Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish.
Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs.
(Menu names can differ by the vendor's interface version and plan.)

HUMAN (PerimeterX)

Subject: Allow verified AI agents (example.com)

An agent-readiness test showed that our bot protection stops automated requests.
So that search and shopping agents can read our product pages, let's allow these bots:
- OAI-SearchBot, ChatGPT-User (OpenAI)
- Claude-SearchBot, Claude-User (Anthropic)
- PerplexityBot, Perplexity-User (Perplexity)
- Googlebot, Bingbot, Applebot

Steps (a HUMAN / PerimeterX signature was seen):
1. Bot Defender → policy: set search and AI agents to "Allow" in the known (good) bots list.
2. Check that these agents aren't shown the "Press & Hold" challenge.

Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish.
Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs.
(Menu names can differ by the vendor's interface version and plan.)

Sucuri

Subject: Allow verified AI agents (example.com)

An agent-readiness test showed that our bot protection stops automated requests.
So that search and shopping agents can read our product pages, let's allow these bots:
- OAI-SearchBot, ChatGPT-User (OpenAI)
- Claude-SearchBot, Claude-User (Anthropic)
- PerplexityBot, Perplexity-User (Perplexity)
- Googlebot, Bingbot, Applebot

Steps (a Sucuri signature was seen):
1. Firewall → Access Control: add the IP ranges the agent vendors publish to the allowlist.
2. Allow by IP verification rather than User-Agent; don't show a challenge page to those IPs.

Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish.
Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs.
(Menu names can differ by the vendor's interface version and plan.)

Example values (example.com, product name, price) are placeholders; replace them with your own.

How to verify it yourself

Allow verified AI bots in your security layer (AI Crawl Control on Cloudflare, bot lists on Akamai/Imperva) and run the test again. You can send the note below to your security team.

Fix it with an AI coding agent

Paste this into Claude Code, Cursor or another coding agent working on your store’s code. Review the change before you deploy it.

You are working on my online store. Specoria’s agent readiness test checks the item below. Fix it so AI shopping agents can read the store and act on it.

## 1. Is bot protection open to agents? (waf_access)
Target: Bot protection stops only malicious traffic; verified search and user agents get the normal page (HTTP 200). Risk-based checks and rate limits (HTTP 429 + Retry-After) replace a CAPTCHA at every step.
Example fix (adapt it to this codebase and platform; example values are placeholders):
```
Subject: Allow verified AI agents (example.com)

An agent-readiness test showed that our bot protection stops automated requests.
So that search and shopping agents can read our product pages, let's allow these bots:
- OAI-SearchBot, ChatGPT-User (OpenAI)
- Claude-SearchBot, Claude-User (Anthropic)
- PerplexityBot, Perplexity-User (Perplexity)
- Googlebot, Bingbot, Applebot

Steps (no bot-protection vendor was detected):
1. In the bot or WAF panel, allow "verified" / known good bots.
2. Add an exception for the search and user agents above; you can keep training bots (GPTBot, ClaudeBot) blocked if you like.
3. On product, cart and policy pages, use rate limits instead of challenge pages.

Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish.
Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs.
(Menu names can differ by the vendor's interface version and plan.)
```
Reference: https://specoria.com/method/checks/waf-access/

## Rules
- First look through the codebase and tell me which files or templates you will change, before editing anything.
- Change only what these checks need, and keep the visible page and the structured data saying the same thing.
- Never invent prices, stock, shipping or return terms, reviews or product identifiers (GTIN): take them from the store’s real data or ask me.
- Don’t block or slow down real visitors and don’t add tracking.
- After deploying, run the free test again at https://specoria.com/ to verify the fix.

Specifications and guidance

The check is based on these open sources; links go to the publisher’s own pages.

Where it sits in the 196-criteria framework

The result of this check counts as evidence for these items of the readiness framework:

  • 1.5 Agent browsers are not blocked
  • 1.6 WAF and bot management do not block legitimate agent traffic

The framework and its source →

Run the free test See the sample report →