How Cloudflare bot rules affect AI agents

If you sit behind Cloudflare, your bot rules are probably the single biggest determinant of whether AI agents can use your site. Two things changed recently that are worth knowing about, and one of them is scheduled for September.

Accurate as of July 2026.

The taxonomy changed on 1 July 2026

Cloudflare now classifies bots by behaviour, and a single bot can carry more than one:

| Behaviour | What it means | |---|---| | Search | Crawling to build search indexes or RAG databases | | Training | Crawling to train or fine-tune models | | Agent | User-directed agents visiting a page on behalf of a human | | Transact | Checkout or other transaction actions on behalf of users | | Data Collection | Price scraping, competitive intelligence, third-party analytics |

There is a second axis describing who operates the bot. Direct means a single narrow operator runs it. Intermediary means an agentic service that many different end users can drive.

That second axis captures a real problem rather than a bureaucratic one. You might be perfectly happy to trust Browserbase as a platform while having no opinion at all about the thousands of individual people driving agents through it. Cloudflare is explicit that this gap exists and is experimenting with forwarding information about the end user.

One naming trap: there is no category literally called "AI Agent". The behaviour is Agent; the legacy category string is AI Assistant. Rules written against a category called "AI Agent" match nothing.

Legacy strings still work in WAF expressions via cf.verified_bot_category: AI Crawler, AI Assistant, AI Search, Page Preview. Cloudflare's own documentation now deprecates AI Search, on the reasoning that there is no longer a meaningful distinction between it and traditional search.

The September default

From 15 September 2026, new domains onboarding to Cloudflare receive updated defaults: bots classified as Training or Agent are blocked on pages that display ads, while Search remains allowed.

Two things to be precise about, because this gets misreported.

It applies to new domains onboarding, not retroactively to every existing free-tier account. Claims that Cloudflare is about to switch this on across the board are not supported by its own changelog.

And it explicitly includes Agent, not just Training. That is the notable part. A default that blocks training crawlers is a licensing stance. A default that also blocks user-directed agents means someone's customer, waiting on an answer, gets turned away by a setting nobody chose.

If you run ad-supported pages and onboard a new domain after that date, check this before you assume the defaults suit you.

Separate the two decisions

The most common mistake is answering one question when there are two:

1. Do I want my content used for training? A licensing and commercial decision. Effects unfold over months.
2. Do I want customers' agents to reach me? A revenue decision. Effects are immediate and invisible.

A single rule that blocks all non-human traffic answers both with a no, usually by accident. Blocking GPTBot costs you presence in a future model. Blocking ChatGPT-User costs you the person trying to buy from you right now.

The new taxonomy is genuinely useful here, because it lets you express the distinction:

# Allow user-directed agents, block bulk training crawls.
(cf.verified_bot_category eq "AI Crawler") -> Block
(cf.verified_bot_category eq "AI Assistant") -> Allow

AI Crawl Control

Cloudflare's AI Crawl Control gives per-crawler Allow, Block or Charge, plus an option to enforce robots.txt at the edge.

That last one deserves a note. Three of the four major operators document that robots.txt may not apply to user-triggered fetches: Google says Google-Agent generally ignores it and publishes no robots.txt token for it, OpenAI says rules may not apply to ChatGPT-User, and Perplexity says the same for Perplexity-User. Anthropic alone applies it uniformly.

Enforcing robots.txt at the edge is therefore doing something quite different from publishing a robots.txt file. The file is a request. Edge enforcement is a rule. If you want a policy actually applied to agents, it has to live here rather than in a text file.

Note also that plan level changes accuracy: the free plan matches on user agent string alone, while paid plans use Bot Management detection. User agent matching is trivially spoofable, so free-tier rules are a stated preference rather than an enforced boundary.

Pay per crawl, which returns HTTP 402 with a crawler-price header, remains in closed beta more than a year after being announced. Worth knowing about, not worth planning around.

Content Signals

Cloudflare's Content Signals Policy adds machine-readable intent to robots.txt comments:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Three signals: search for indexing, ai-input for feeding content to a model at answer time, ai-train for training. Absent means neither granted nor restricted.

The important limitation: there is no agent signal. Content Signals governs what may be done with your content, not whether an agent may act on your site. It does not solve the problem this article is about.

What to do

Find out what you currently do. Most people have not looked at these rules since they were set. Check what happens to AI Assistant traffic today.

Decide the two questions separately, and write rules that reflect both answers.

Do not rely on robots.txt for agents. Whatever you decide, enforce it at the edge, because that is the only place it will actually apply.

Diarise 15 September 2026 if you have new domains coming and ad-supported pages.

Verify from the outside. Rules do not always behave the way the dashboard suggests. Request your own site the way an agent would and see what comes back. If a check of your own profile comes back challenged or rate limited, that is your answer: a real agent hits the same wall.

Read next

• AI crawlers vs AI agents — why the distinction drives all of this
• Why agent access breaks silently — why a blocked agent never reaches your attention
• Agent traffic in your analytics — spotting what does get through