--- title: "Is bot protection open to agents? · Specoria" description: "Is bot protection open to agents?: An agent that sees a 403 or a challenge page can’t compare the store; browser agents (ChatGPT agent, Comet) get stuck too." source_url: "https://specoria.com/method/checks/waf-access/" lang: "en" --- [← All checks](https://specoria.com/method/checks/) waf_access # Is bot protection open to agents? Why it matters An agent that sees a 403 or a challenge page can’t compare the store; browser agents (ChatGPT agent, Comet) get stuck too. **Category** Access **Weight** 8 of 100 points **Critical blocker** Yes. If it fails, the score is capped at 59; with two or more blockers, at 39. **Applies to** Online stores · Hotels and lodging · Service businesses **Effort** 1-2 hours · infrastructure or CDN admin **Score model** v2.4 ## How we measure it We request the homepage and the product page with ordinary browser headers. A hard refusal (e.g. HTTP 403) fails the check and is a critical blocker. A challenge page or CAPTCHA is a “possible block”: a partial pass, not a critical blocker, since we can’t see from outside what verified agents get. A rate limit (HTTP 429) isn’t counted as a block: we retry after the Retry-After wait; if the page opens it passes, if it’s still limited the check is not verifiable and doesn’t count. If the homepage is blocked from Cloudflare’s network but opens from a second network, it’s a partial pass. If only the one request that identifies itself as SpecoriaBot is refused, that’s also a “possible block” (partial pass): we can’t send requests from verified bot IPs. On hotel and service sites this check is measured on the homepage instead of a product page. ### What the result page says These are the sentences in the test result; “…” is replaced by what we found on your site. - Passed The homepage and product page responded normally to an automated request; we didn’t hit a challenge wall. - Passed The first request hit a rate limit (……); after a short wait the page opened normally. A rate limit isn’t a block: answering HTTP 429 with a Retry-After header is the right behaviour for agents. - Partial Our scan from Cloudflare’s network was refused (……), but … opened normally from a second data center and the scan was completed from there. The block is specific to Cloudflare traffic; most agents come from other networks, though browser agents running on Cloudflare’s infrastructure may get stuck. - Partial The page opened with a browser identity, but a request that identified itself as a bot (SpecoriaBot) was refused (……). This is a possible block: a rule that blocks bots by identity can also stop AI agents such as OAI-SearchBot; real agents come from verified IPs and may be treated differently. - Partial … showed our scan a challenge page (……). This is a possible block: agent crawlers that don’t run JavaScript can’t get past it. If verified AI bots are allowed they may see the normal page; we can’t verify that from outside. - Partial … showed our scan a challenge page both from Cloudflare’s network and from a second data center (……). This is a possible block: agent crawlers that don’t run JavaScript can’t get past it. If verified AI bots are allowed they may see the normal page; we can’t verify that from outside. - Partial … showed our scan a challenge page (……). This is a possible block: an agent may leave without seeing the product. If verified AI bots are allowed they may see the page; we can’t verify that from outside. - Failed … turned our scan away (……). The site’s security layer stops automated requests, and shopping agents may hit the same wall. - Failed … turned our scan away (……). An agent leaves without seeing the product. - Failed … turned our scan away both from Cloudflare’s network and from a second data center (……). The site’s security layer blocks data-center traffic in general, so shopping agents will most likely hit the same wall. ## How to fix it **Target:** Bot protection stops only malicious traffic; verified search and user agents get the normal page (HTTP 200). Risk-based checks and rate limits (HTTP 429 + Retry-After) replace a CAPTCHA at every step. ### Example code General `Subject: Allow verified AI agents (example.com) An agent-readiness test showed that our bot protection stops automated requests. So that search and shopping agents can read our product pages, let's allow these bots: - OAI-SearchBot, ChatGPT-User (OpenAI) - Claude-SearchBot, Claude-User (Anthropic) - PerplexityBot, Perplexity-User (Perplexity) - Googlebot, Bingbot, Applebot Steps (no bot-protection vendor was detected): 1. In the bot or WAF panel, allow "verified" / known good bots. 2. Add an exception for the search and user agents above; you can keep training bots (GPTBot, ClaudeBot) blocked if you like. 3. On product, cart and policy pages, use rate limits instead of challenge pages. Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish. Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs. (Menu names can differ by the vendor's interface version and plan.)` Cloudflare `Subject: Allow verified AI agents (example.com) An agent-readiness test showed that our bot protection stops automated requests. So that search and shopping agents can read our product pages, let's allow these bots: - OAI-SearchBot, ChatGPT-User (OpenAI) - Claude-SearchBot, Claude-User (Anthropic) - PerplexityBot, Perplexity-User (Perplexity) - Googlebot, Bingbot, Applebot Steps (a Cloudflare signature was seen): 1. Security → Bots: allow verified bots ("Allow verified bots" in Super Bot Fight Mode). 2. AI Crawl Control: set OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User to "Allow". 3. If needed, a custom WAF rule: (cf.verified_bot_category in {"AI Search" "AI Assistant" "Search Engine Crawler"}) → Skip. Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish. Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs. (Menu names can differ by the vendor's interface version and plan.)` Akamai `Subject: Allow verified AI agents (example.com) An agent-readiness test showed that our bot protection stops automated requests. So that search and shopping agents can read our product pages, let's allow these bots: - OAI-SearchBot, ChatGPT-User (OpenAI) - Claude-SearchBot, Claude-User (Anthropic) - PerplexityBot, Perplexity-User (Perplexity) - Googlebot, Bingbot, Applebot Steps (an Akamai signature was seen): 1. Bot Manager → Akamai-categorized known bots: set the search engine and AI search/assistant categories to "Allow". 2. Make sure no challenge or tarpit action applies to those categories. 3. You can decide separately on training bots; keep search and user agents open. Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish. Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs. (Menu names can differ by the vendor's interface version and plan.)` DataDome `Subject: Allow verified AI agents (example.com) An agent-readiness test showed that our bot protection stops automated requests. So that search and shopping agents can read our product pages, let's allow these bots: - OAI-SearchBot, ChatGPT-User (OpenAI) - Claude-SearchBot, Claude-User (Anthropic) - PerplexityBot, Perplexity-User (Perplexity) - Googlebot, Bingbot, Applebot Steps (a DataDome signature was seen): 1. Bot Protection → allowlist (good bots): set OpenAI, Anthropic and Perplexity search and user agents to "Allow". 2. Check that the CAPTCHA rule is off for verified bots. Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish. Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs. (Menu names can differ by the vendor's interface version and plan.)` Imperva `Subject: Allow verified AI agents (example.com) An agent-readiness test showed that our bot protection stops automated requests. So that search and shopping agents can read our product pages, let's allow these bots: - OAI-SearchBot, ChatGPT-User (OpenAI) - Claude-SearchBot, Claude-User (Anthropic) - PerplexityBot, Perplexity-User (Perplexity) - Googlebot, Bingbot, Applebot Steps (an Imperva signature was seen): 1. Bot Management → client classification: allow search bots and AI agents as good bots. 2. Make sure Security Rules don't apply a challenge (JavaScript/CAPTCHA) to those classes. Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish. Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs. (Menu names can differ by the vendor's interface version and plan.)` HUMAN (PerimeterX) `Subject: Allow verified AI agents (example.com) An agent-readiness test showed that our bot protection stops automated requests. So that search and shopping agents can read our product pages, let's allow these bots: - OAI-SearchBot, ChatGPT-User (OpenAI) - Claude-SearchBot, Claude-User (Anthropic) - PerplexityBot, Perplexity-User (Perplexity) - Googlebot, Bingbot, Applebot Steps (a HUMAN / PerimeterX signature was seen): 1. Bot Defender → policy: set search and AI agents to "Allow" in the known (good) bots list. 2. Check that these agents aren't shown the "Press & Hold" challenge. Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish. Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs. (Menu names can differ by the vendor's interface version and plan.)` Sucuri `Subject: Allow verified AI agents (example.com) An agent-readiness test showed that our bot protection stops automated requests. So that search and shopping agents can read our product pages, let's allow these bots: - OAI-SearchBot, ChatGPT-User (OpenAI) - Claude-SearchBot, Claude-User (Anthropic) - PerplexityBot, Perplexity-User (Perplexity) - Googlebot, Bingbot, Applebot Steps (a Sucuri signature was seen): 1. Firewall → Access Control: add the IP ranges the agent vendors publish to the allowlist. 2. Allow by IP verification rather than User-Agent; don't show a challenge page to those IPs. Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish. Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs. (Menu names can differ by the vendor's interface version and plan.)` Example values (example.com, product name, price) are placeholders; replace them with your own. ### How to verify it yourself Allow verified AI bots in your security layer (AI Crawl Control on Cloudflare, bot lists on Akamai/Imperva) and run the test again. You can send the note below to your security team. ### Fix it with an AI coding agent Paste this into Claude Code, Cursor or another coding agent working on your store’s code. Review the change before you deploy it. `You are working on my online store. Specoria’s agent readiness test checks the item below. Fix it so AI shopping agents can read the store and act on it. ## 1. Is bot protection open to agents? (waf_access) Target: Bot protection stops only malicious traffic; verified search and user agents get the normal page (HTTP 200). Risk-based checks and rate limits (HTTP 429 + Retry-After) replace a CAPTCHA at every step. Example fix (adapt it to this codebase and platform; example values are placeholders): ``` Subject: Allow verified AI agents (example.com) An agent-readiness test showed that our bot protection stops automated requests. So that search and shopping agents can read our product pages, let's allow these bots: - OAI-SearchBot, ChatGPT-User (OpenAI) - Claude-SearchBot, Claude-User (Anthropic) - PerplexityBot, Perplexity-User (Perplexity) - Googlebot, Bingbot, Applebot Steps (no bot-protection vendor was detected): 1. In the bot or WAF panel, allow "verified" / known good bots. 2. Add an exception for the search and user agents above; you can keep training bots (GPTBot, ClaudeBot) blocked if you like. 3. On product, cart and policy pages, use rate limits instead of challenge pages. Identity: don't trust the User-Agent alone; verify against the IP ranges or reverse DNS the vendors publish. Slow down abusive traffic with rate limits (HTTP 429 + Retry-After) instead of CAPTCHAs. (Menu names can differ by the vendor's interface version and plan.) ``` Reference: https://specoria.com/method/checks/waf-access/ ## Rules - First look through the codebase and tell me which files or templates you will change, before editing anything. - Change only what these checks need, and keep the visible page and the structured data saying the same thing. - Never invent prices, stock, shipping or return terms, reviews or product identifiers (GTIN): take them from the store’s real data or ask me. - Don’t block or slow down real visitors and don’t add tracking. - After deploying, run the free test again at https://specoria.com/ to verify the fix.` ## Specifications and guidance - [OpenAI: crawlers and user agents ↗](https://platform.openai.com/docs/bots) The check is based on these open sources; links go to the publisher’s own pages. ## Where it sits in the 196-criteria framework The result of this check counts as evidence for these items of the readiness framework: - `1.5` Agent browsers are not blocked - `1.6` WAF and bot management do not block legitimate agent traffic [The framework and its source →](https://specoria.com/method/#criteria) [← PreviousSearch and user agents allowed (robots.txt)](https://specoria.com/method/checks/bots-search/)[Next →Product name and price readable without JavaScript](https://specoria.com/method/checks/no-js/) [Run the free test](https://specoria.com/method/checks/waf-access/#test) [See the sample report →](https://specoria.com/reports/sample-report/) ## Related - [Search and user agents allowed (robots.txt)](https://specoria.com/method/checks/bots-search/) - [Product name and price readable without JavaScript](https://specoria.com/method/checks/no-js/) - [Agent protocol map: UCP, ACP, AP2, MCP, WebMCP](https://specoria.com/agent-protocols/) What each protocol does and what the free test checks - [Scan API, MCP server and GitHub Action](https://specoria.com/developers/) Run the free test from your own scripts or CI FREE TEST ## Test your store now Enter your store’s address: 27 checks on your public pages, the way a shopping agent reads them. The free test runs in your browser and needs JavaScript. [Request the free deep audit by email instead →](https://specoria.com/#contact) Your result opens on its own page. Want the full picture? The free deep audit and panel come next, by email.