AI crawlers vs AI agents: why the difference decides your bot rules

A crawler fetches pages so a system can index or train on them later. Nobody is waiting. An agent fetches a page because a person asked it to do something right now, and that person is watching a spinner.

The distinction sounds academic until you look at your bot rules, which were almost certainly written when only the first kind existed.

Accurate as of July 2026.

The practical difference

| | Crawler | Agent | |---|---|---| | Triggered by | A schedule | A person, just now | | Waiting | Nobody | Someone | | Volume | High, broad | Low, targeted | | robots.txt | Authoritative | Often treated as not applying | | Blocking it costs you | Presence in an index | A customer, immediately |

That last row is the one that matters. Blocking GPTBot is a content licensing decision with effects that show up over months. Blocking ChatGPT-User is turning away someone who is trying to buy from you this minute. They can look almost identical in your logs and they are not remotely the same event.

The robots.txt split, in operators' own words

Here is where it gets genuinely awkward. The major operators do not agree on whether robots.txt governs user-triggered fetches, and several say plainly that it does not.

OpenAI distinguishes its crawler from its user fetcher. For ChatGPT-User it states that because the actions are initiated by a user, robots.txt rules may not apply. For GPTBot, robots.txt is authoritative.

Google goes further with Google-Agent, introduced in March 2026: because the fetch was requested by a user, these fetchers generally ignore robots.txt. Google publishes no robots.txt token for it at all, so it is not a control surface.

Perplexity documents that since a user requested the fetch, Perplexity-User generally ignores robots.txt.

Anthropic is the outlier, and deliberately so. It applies robots.txt uniformly across all of its bots, including the user-triggered Claude-User.

Three of the four major operators have carved out user-initiated traffic. If your entire agent strategy is a robots.txt entry, it is not doing what you think it is doing for most of them.

The reasoning is not unreasonable, incidentally. robots.txt governs automated crawling. When a person explicitly asks software to go and look at a specific page, the operators argue that is closer to that person visiting than to a crawl. You do not have to agree, but you do have to plan around it.

Most agents cannot be identified at all

The crawler model was simple: a declared user agent token plus published IP ranges. Match the string, verify the address, done.

That model has largely broken down for agents, and not because anyone is being evasive. It is architectural.

Claude for Chrome, Copilot Actions and Perplexity's Comet browser all drive the user's own browser, with the user's own IP address, cookies and session. There is nothing distinctive for your server to match on, because there is nothing distinctive to send. The traffic is indistinguishable from the person, because in a meaningful sense it is the person.

Microsoft has published no agent user agent token whatsoever. Anthropic publishes bare tokens but no full user agent strings for Claude for Chrome. When we compiled our agents directory, five of the nine agents we tracked published no user agent at all.

A practical consequence: CopilotBot circulates widely in third-party bot directories and appears in no Microsoft source. If it is in your filters, it is filtering nothing.

What replaced it: signed requests

The emerging answer is cryptography rather than string matching. Under Web Bot Auth, an agent signs its HTTP requests using RFC 9421 HTTP Message Signatures and publishes its public keys at a well-known location:

GET https://<signing-host>/.well-known/http-message-signatures-directory

The request carries a Signature-Agent header pointing at that host, plus Signature and Signature-Input headers with tag="web-bot-auth". Your origin fetches the key directory, matches the keyid against a JWK thumbprint, and verifies the Ed25519 signature. Now you know who is calling, and a copied user agent string proves nothing.

This is real and deployed, but thinly. When we fetched signing directories directly in July 2026, four were live and valid: Google (at agent.bot.goog, five keys), OpenAI (at chatgpt.com, one key), Browserbase and You.com. Anthropic, Perplexity and Microsoft published none at their canonical hosts.

Two traps if you implement verification yourself:

• Check the content type, not the status code. Several hosts return 200 at that path while serving an ordinary HTML page. A status check gives you false positives.
• A directory can be empty. Shopify serves a valid directory with the correct content type containing zero keys. The endpoint exists; the capability is not active. Serving a directory is not the same as being verifiable.

Also note Google's own caveat: only a subset of Google-Agent requests are signed today, and Google recommends continuing to use IP ranges and user agent matching alongside signatures. Signature-only logic will drop legitimate traffic.

How Cloudflare now models it

If you are behind Cloudflare, its taxonomy changed on 1 July 2026 and is worth understanding because it is what your rules can actually express.

Bots are now classified by behaviour, and one bot can have several:

• Search — crawling to build search indexes or RAG databases
• Training — crawling to train or fine-tune models
• Agent — user-directed agents visiting a page on behalf of a human
• Transact — checkout or other transaction actions on behalf of users
• Data Collection — price scraping, competitive intelligence, third-party analytics

There is a second axis: Direct means one narrow operator runs it; Intermediary means an agentic service that many different end users can drive. That second one captures a genuine problem. You might trust Browserbase as a platform while having no view at all on the thousands of individual people driving agents through it. A valid signature tells you the request came through Browserbase. It does not tell you who was steering.

Note there is no category literally called "AI Agent". The behaviour is Agent; the legacy category string is AI Assistant.

One date to diarise: from 15 September 2026, new domains onboarding to Cloudflare receive defaults that block traffic classified as Training or Agent on pages that display ads. That is scoped to new domains, not a retroactive sweep, but it tells you the direction.

What to actually do

Separate the two decisions. "Do I want my content used for training?" and "do I want customers' agents to be able to reach me?" are different questions with different answers. A single rule that blocks all non-human traffic answers both with a no, probably by accident.

Stop relying on robots.txt alone for agents. Three of four major operators have said it may not apply to user-triggered fetches. Whatever you decide, decide it at the edge where it will actually be enforced.

Do not filter on invented tokens. If a user agent string does not appear in an operator's own documentation, it came from somebody's blog. Our agents directory records what each operator actually publishes, including where the honest answer is "nothing".

Watch for the failure you cannot see. A blocked agent produces no error on your side. It is a 403 in your logs that looks exactly like the bot mitigation you asked for, and a person somewhere who did not get what they wanted. If you want to know whether your own site is reachable, run a check that requests it the way an agent would.

Read next

• How agents see your website — what happens between arrival and action
• What is UCP? — how agents discover what you sell before they visit
• The agents directory — who is visiting, and which ones you can verify