How agents see your website

Your site was built for someone with eyes, a browser and a rough idea of how shops work. An agent has none of those things. It has an HTTP client, a model, and an instruction from someone who has already stopped paying attention.

Understanding what it goes through is the fastest way to see why agent failures are so hard to notice.

Accurate as of July 2026.

The four stages

1. DISCOVERY   Is there a machine-readable description of this business?
                  GET /.well-known/ucp
                       |
2. ACCESS      Can I actually fetch the pages?
                  bot rules, rate limits, challenges
                       |
3. COMPREHENSION   What is on this page and what can I do?
                  WebMCP tools, else infer from markup
                       |
4. ACTION      Complete the task
                  search, filter, add to cart, check out

Each stage can fail independently. A failure at stage two looks like bot mitigation working. A failure at stage three looks like nothing at all.

Stage one: discovery

Before loading any page, an agent may check whether you describe yourself in a format it can read. For commerce that means a UCP profile at /.well-known/ucp.

If it finds one, it knows what you sell, which capabilities you support and how payment works, without parsing a single page of HTML. If it does not, it falls back to reading your site like a slightly confused visitor.

Where this fails: the profile exists but is served behind a redirect. The specification says agents must not follow redirects here, and they do not. Your apex redirects to www, the profile only answers on one host, and to half of callers you simply have no integration. In a browser you would never notice, because your browser follows the redirect without comment.

Stage two: access

Now it needs to fetch pages, and this is where the largest share of silent failures live.

Your bot rules were written to keep scrapers out. An agent arrives looking, from the edge's point of view, quite a lot like a scraper: an unfamiliar client, possibly an unfamiliar IP range, requesting pages quickly. It gets challenged or blocked.

What makes this so easy to miss is that everything worked exactly as configured. There is no error. Your dashboards show bot mitigation doing its job. The only trace is a 403 among thousands of legitimate 403s, and a person who did not get an answer.

Three specifics worth knowing:

robots.txt may not apply. Google, OpenAI and Perplexity all document that user-triggered fetches may ignore robots.txt. Anthropic alone applies it uniformly. If your agent policy is a robots.txt entry, it is not being enforced the way you assume.

Rate limits catch bursts. An agent comparing five products may request five pages in two seconds. That is not abusive, but it can look like it.

Defaults are shifting. From September 2026, new domains onboarding to Cloudflare get defaults that block agent-classified traffic on pages showing ads.

There is a diagnostic worth internalising here: if a check of your own site comes back rate limited or challenged, that is a finding, not a non-result. A real agent hits exactly the same wall. It is why our checker reports that case as indeterminate rather than pretending it means you have no integration.

Stage three: comprehension

The agent has your HTML. Now it has to work out what is on the page and what it can do.

Without WebMCP, this is inference. It reads the DOM and the accessibility tree, looks at labels, roles, headings and form fields, and forms a theory: this input is probably a search box, this button probably adds to a basket, this number is probably the price.

The theory is often right. When it is wrong, it is wrong in ways you would not predict:

• A price rendered in an image, or assembled by script after load
• A "size" control built from styled divs with no roles
• An input labelled only by a placeholder and an icon
• Three buttons that all say "Select"
• Content that only appears after an interaction the agent does not know to perform

Nothing here is broken. It is all perfectly usable by a person, which is the point. Visual design does a lot of work that markup does not, and the agent only gets the markup.

The single highest-leverage thing you can do at this stage costs nothing new: semantic HTML and real accessibility markup. Proper labels, roles, headings and states. The work you would do to support a screen reader is, almost exactly, the work that makes your site legible to an agent. If you have been putting off an accessibility audit, it now has a second business case.

WebMCP replaces inference with declaration. Rather than hoping the agent identifies your search box, you declare a searchProducts tool with a typed schema and a plain description. Certainty instead of a good guess.

Stage four: action

Finally it tries to do the thing: add to basket, apply a filter, complete a purchase.

Where this fails: multi-step flows that assume a person. A modal that must be dismissed. A cookie banner overlaying the button. A control that only appears on hover. A checkout that depends on state built up across three pages. Each is trivial for a human and a wall for software.

This is also where the highest-stakes failure sits, because a person is at the end of it waiting for a purchase to complete.

Why you never hear about any of this

Every failure above produces a normal-looking server response.

A blocked agent is a 403 you configured. An unfetchable profile is a 301 you set up on purpose. A misread page is a 200 with correct HTML. An abandoned checkout is one abandoned checkout among many.

Compare this to a broken payment provider, where you get alerts, error rates and probably a phone call. Agent failures generate none of that, because from your infrastructure's point of view nothing went wrong. Someone just left.

That asymmetry is the whole reason this needs deliberate checking rather than monitoring. Your existing monitoring is watching for errors, and there are no errors.

Seeing it for yourself

Four things you can do today, in rough order of effort:

1. Fetch your own profile the way an agent would, without following redirects, on both apex and www:

   curl -sI -H "Accept: application/json" https://yourdomain.com/.well-known/ucp
   

   Check the status and the Cache-Control header, not just the body.

2. Read your own product page as text. Disable CSS and images, or run it through a text-mode browser. Anything you cannot work out from what remains, an agent probably cannot either.

3. Audit the accessibility of one key flow. Whatever you fix will improve agent comprehension at the same time.

4. Run a check and let something else do stages one and two for you, with the fixes attached.

Read next

• AI crawlers vs AI agents — why the two need different rules at the edge
• What is UCP? — the discovery layer in detail
• What is WebMCP? — replacing inference with declared tools