Specoria
Menu
TR Free test

InsightsInfrastructure

A bot policy by page type: what to open to AI agents and what to keep closed

Product and policy pages open, cart and checkout for verified agents, admin closed. How to set it in robots.txt, Content Signals and Cloudflare.

By 8 min read

Open your CDN's bot report and you will probably find a long list of AI names: crawlers that train models, bots that build search indexes, and fetchers that open a page because one person asked a question. The tempting response is one switch that blocks them all or lets them all in. Neither fits how a store works.

A product page and an admin login are not the same kind of page, and a training crawler and an agent buying for a shopper are not the same kind of visitor. In this article, we set out a bot policy by page type and express it with the tools available in October 2026: robots.txt, Content Signals, Cloudflare's AI controls and signed agents. For identity basics, see Bot or buyer?.

Start with the page, not the bot

Four zones cover most stores:

  • Product, category and policy pages: open. This is where an agent decides whether your store makes the shortlist: price, stock, variants, shipping costs, the return window. Search crawlers and user-triggered fetchers should reach them without a challenge.
  • Cart and checkout: open to agents acting for a shopper, with verification. No crawler needs to index a cart, but an agent sent to buy something has to get through it. Let verified agents through without a CAPTCHA, and keep tighter checks for unidentified automation at the payment step.
  • Account pages: behind sign-in. The control is the login itself, not a bot rule.
  • Admin and internal tools: closed. No bot of any kind has business there.

robots.txt: three groups, three answers

AI companies now split their crawlers by purpose, so you can answer each group separately.

  • Search crawlers. OpenAI's OAI-SearchBot surfaces sites in ChatGPT's search features; OpenAI says sites that opt out of it will not be shown in ChatGPT search answers. Anthropic's Claude-SearchBot indexes content for Claude's search results. Treat them like Googlebot and Bingbot: allow product, category and policy pages.
  • User-triggered fetchers. ChatGPT-User, Claude-User and Google-Agent open pages because a person asked. Anthropic says site owners can control Claude-User through robots.txt. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, because a user started the action. Google says its user-triggered fetchers "generally ignore robots.txt rules". For this group, robots.txt is a preference, not a gate.
  • Training crawlers. GPTBot and ClaudeBot collect content that may be used to train models. Google-Extended is not a separate crawler but a robots.txt token that controls whether content Google crawls is used for Gemini training and grounding. Google says it does not affect inclusion or ranking in Google Search. Training is a business choice; refusing it need not cost you search.

In practice, a store's file allows product, category and policy paths, adds Disallow lines for the cart, checkout and account paths your platform really uses (/cart or /sepet, /checkout, /account), and adds User-agent: GPTBot, User-agent: ClaudeBot or User-agent: Google-Extended groups only if you opt out of training. One limit matters more than any line: as Cloudflare puts it, anyone can publish a robots.txt, but it cannot identify who is crawling or stop a crawler that ignores it. Don't rely on it to protect your admin.

Content Signals: what may happen to a page after it is read

robots.txt says who may fetch which path, not what happens to the content afterwards. Cloudflare's Content Signals Policy, published in September 2025 under a CC0 license, adds that layer with three signals:

  • search: building a search index and showing results such as links and short excerpts. Cloudflare's definition excludes AI-generated search summaries.
  • ai-input: putting content into AI models at answer time, for example retrieval or grounding for a generative answer.
  • ai-train: training or fine-tuning models.

A line such as Content-Signal: search=yes, ai-train=no allows search, refuses training and expresses no preference on ai-input. For a store, the middle signal deserves thought. When a shopper asks an assistant which kettle can arrive by Friday, the answer is built from ai-input; ai-input=no asks AI systems not to use your product pages for exactly that. Cloudflare also notes that content signals are preferences, not technical countermeasures, and some companies may ignore them.

Cloudflare's AI controls after September 15, 2026

If your store runs behind Cloudflare, two 2026 changes matter. On July 1, Cloudflare gave all customers, including the Free plan, three AI behaviors to control separately: Search (indexing content to answer questions later), Agent (automated activity acting in real time for a person, such as chat fetch bots and browser-use agents) and Training. For each one you can block on all pages, block only on pages that display ads, or not block.

On September 15, it addressed mixed-use crawlers, single crawlers that do both search and training, such as Googlebot, Bingbot and Applebot:

  • A new Training setting, Disallow AI Training. It publishes a no-training preference in robots.txt, keeps mixed-use crawlers that Cloudflare classes as "Accountable" allowed for search, and blocks other training crawlers, including the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI.
  • Block now reaches mixed-use crawlers. Block and Block on pages with ads now apply to Googlebot, Bingbot and Applebot, so they affect search as well as training. To stop training and keep search, Cloudflare says to use Disallow AI Training, not Block.
  • Existing settings carried over. Cloudflare says that in almost every case customers need to do nothing. Mixed-use crawlers are not blocked by default; they are blocked only if you choose Block.
  • New domains get a preset. Sites that don't earn money from ads get Search, Training and Agent set to Allow. Ad-supported sites get Disallow AI Training, with Agent blocked on pages with ads. Either can be changed at any time.

One gap remains: Microsoft's support for a robots.txt no-training preference in Bing is targeted for early 2027; until then, Disallow AI Training does not pass that preference to Bing. For a store, the practical rule is simple: if you opt out of training, choose Disallow AI Training, not Block, or your product pages can drop out of Google and Bing search too.

Cart and checkout: verify instead of guessing

Anyone can copy a name into a user-agent header, so the cart and checkout zone needs identity, not a list of names. Web Bot Auth lets an agent sign each request with its own key, using HTTP message signatures and a published key directory, both described in IETF drafts. Since July 1, 2026, Cloudflare treats signed agents as Verified bots, labeled either Direct (one operator sends the requests) or Intermediary (an agent service that many end users can operate). Its classification also includes a Transact behavior: checkout or other transactions on behalf of users. Google says it is experimenting with Web Bot Auth for Google-Agent, using the https://agent.bot.goog identity.

Cloudflare notes the catch with intermediaries: you may trust the operator without trusting every end user behind it. Verification tells you who is driving, not whether the order is good, so keep your fraud checks. Just don't put a CAPTCHA in front of a verified agent on its way to checkout.

A checklist for this week

  1. Map your four zones with the real paths your platform uses.
  2. Rewrite robots.txt by group: product and policy pages open to search crawlers and fetchers, cart, checkout and account closed to crawlers, training decided separately.
  3. Add a matching Content-Signal line, and think twice before ai-input=no.
  4. On Cloudflare, allow Search and Agent; for training, use Disallow AI Training rather than Block. Check the preset on any recently added domain.
  5. Let verified agents skip challenges on product, cart and checkout pages; keep CAPTCHAs for unidentified traffic at payment.
  6. Protect admin with authentication and network rules, never with robots.txt alone.
  7. Read a week of logs. AI agent requests to product or checkout pages that got a 403 or 429 are sales you never saw.

The bottom line

The real question isn't whether to let AI in. It's which visitor gets which page. Product and policy pages are your shop window, readable by any honest crawler or agent. Cart and checkout are the counter, where you want to see who is paying and for whom. Accounts and admin stay behind doors a bot rule was never meant to guard.

Specoria checks this from the outside: whether your robots.txt lets search and user agents reach product and policy pages, whether bot protection stops automated requests, and whether a real browser agent gets from product page to checkout. Start with a free readiness test.

Sources