Directory
AI agents
Agents act on a person's behalf. They browse, fill forms, and increasingly buy. That makes them a
different problem from crawlers, and most of them cannot be identified the way a crawler can.
9
agents tracked, across 8 operators
4
can be cryptographically verified today
5
publish no user agent at all
Signing directories were checked by direct fetch on 26 July 2026, validating the JWKS content type rather
than a 200 status, because several hosts return 200 with an ordinary HTML page at that path.
Why most of this table is empty
The crawler model was a declared token plus a published IP list. That model has broken down for agents.
Claude for Chrome, Copilot Actions and Perplexity's Comet all drive the user's own browser, with the
user's address, cookies and session. There is nothing for your server to match on, because there is
nothing to match. Any directory that prints a user agent string for those products invented it.
The replacement is request signing. An agent signs its requests and publishes its public keys at a
well known URL, so your origin can verify the traffic instead of trusting a string anyone can copy.
That is what the verified column below measures.
User-directed agent
Visits and acts on a page on behalf of a person who is waiting for the result.
- User agent
-
Google-Agent
- Respects robots.txt
-
No, documented to ignore it
Used by agents hosted on Google infrastructure to navigate the web and perform actions on a user's request.
Details and how to verify
- User agent
-
ChatGPT-User
- Respects robots.txt
-
Operator says it may not apply
Fetches a page in response to a specific user request inside ChatGPT.
Details and how to verify
- User agent
-
None published
- Respects robots.txt
-
Not applicable
Publishes a Web Bot Auth key directory for its agent traffic.
Details and how to verify
- User agent
-
None published
- Respects robots.txt
-
Not documented
Runs inside the user's own Chrome profile and session, acting on the page in front of them.
Details and how to verify
- User agent
-
None published
- Respects robots.txt
-
Not documented
Drives the user's own Edge browser with the user's cookies, selecting, scrolling and typing the way the person would.
Details and how to verify
Assistant fetch
Retrieves a page in real time to answer a question a user just asked.
- User agent
-
Claude-User
- Respects robots.txt
-
Yes, uniformly
Retrieves a page because a user asked Claude about it. Distinct from ClaudeBot, which crawls for training.
Details and how to verify
- User agent
-
Perplexity-User
- Respects robots.txt
-
No, documented to ignore it
Fetches a page to answer a question a user just asked.
Details and how to verify
Agent infrastructure
Hosted browser platforms that other people's agents run on.
- User agent
-
None published
- Respects robots.txt
-
Not applicable
Hosted browser infrastructure that other people's agents run on. Many different end users drive the same platform.
Details and how to verify
Transacting agent
Completes checkout or another transaction on behalf of a user.
- User agent
-
None published
- Respects robots.txt
-
Not applicable
Agent traffic originating from Shopify commerce surfaces.
Details and how to verify
Looking for crawlers rather than agents? The
AI crawlers directory
covers the bots that fetch pages for training and indexing.