Guides to the standards that decide whether AI agents can use your website.
A crawler fetches pages for an index. An agent acts on behalf of a waiting person. They treat robots.txt differently, they are verified differently, and blocking one is not the same as blocking the other.
UCP is an open standard that lets an AI agent discover what your business sells and how to buy it. What it is, how discovery works, and the two rules most implementations get wrong.
WebMCP lets your site declare typed tools an AI agent can call directly, instead of making it interpret your markup. What it is, where the Chrome trial has got to, and the rename that broke older guides.
Understand the AI crawler bots visiting your website, including GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended. Learn how to configure robots.txt for AI crawlers.
Learn about llms.txt, the emerging standard that helps AI models understand your brand. Discover what to include, how it differs from robots.txt, and how to create one for your site.
AI crawlers are automated bots operated by AI companies — like GPTBot, ClaudeBot, and PerplexityBot — that scan and index web content for training data and real-time retrieval.
JSON-LD (JavaScript Object Notation for Linked Data) is a structured data format used to embed machine-readable metadata in web pages, helping AI systems and search engines understand page content.
LLMs.txt is a proposed standard file that websites can host to provide AI systems with structured information about their brand, products, and positioning.
Robots.txt for AI refers to using the robots.txt file to specifically control which AI crawlers can access your website content for training and retrieval purposes.
Structured data is machine-readable metadata embedded in web pages that explicitly describes the content's meaning, type, and attributes using standardized vocabularies like Schema.org.
A crawler fetches pages for an index. An agent acts on behalf of a waiting person. They treat robots.txt differently, they are verified differently, and blocking one is not the same as blocking the other.
What actually happens when an AI agent arrives at your site: the four stages from discovery to action, where each one fails, and why none of the failures reach your error logs.
Three standards, three different jobs. UCP describes your business, ACP moves a transaction, AP2 proves the user authorised it. What each actually does, and which ones overlap.
UCP is an open standard that lets an AI agent discover what your business sells and how to buy it. What it is, how discovery works, and the two rules most implementations get wrong.
WebMCP lets your site declare typed tools an AI agent can call directly, instead of making it interpret your markup. What it is, where the Chrome trial has got to, and the rename that broke older guides.
The request goes to an AI assistant, and the assistant does the visiting. When it cannot work out what a site offers it uses another one, and no report anywhere records that it happened.
If you sell on Shopify you probably already have a UCP profile you never created. What it contains, the one thing that is commonly wrong with it, and which parts are yours to fix.
A practical guide to declaring WebMCP tools: what to expose, how to write descriptions an agent can actually choose between, and the security boundary people forget.
A checklist for confirming your UCP profile actually works: transport, discovery, document structure and the mistakes that only show up when an agent tries it rather than a browser.
A UCP profile advertises that you support checkout. Making that true is a different job. The patterns that stop an agent completing a purchase, and what to do about them.
WebMCP fails silently: rename the entry point and your page registers nothing, with no error anywhere. How to catch that in a build instead of in production.
Serving a UCP profile from a Laravel application: the route, the headers the spec requires, handling apex and www, and a test that fails the build when discovery breaks.
How to tell agent traffic from crawlers and people in your logs, why most of it is invisible by design, and what is actually worth measuring when the interesting event is a non-event.
Cloudflare replaced its bot taxonomy in July 2026 and is changing defaults in September. What the new behaviour classes mean, and how to allow customers' agents without opening the door to scrapers.
UCP versions are dates, not numbers, so the version string never tells you whether a release breaks you. What has changed so far, and how to keep up without reading every diff.
Every way an agent fails on your site produces a normal-looking response. No errors, no alerts, no spike on a dashboard. Why your existing monitoring cannot see it, and what to watch instead.
Reading about it only gets you so far. Run a free check and find out in a moment, with no account and no developer needed.