The full 📝Cloudflare Agent Readiness check list grouped by implementation effort — for the engineer or SEO who has a failing scan open and wants to know what to actually serve.
Ship the quick wins
- **📝robots.txt** — Serve a valid, crawler-readable
/robots.txt. 78% of the top 200,000 domains have one, but nearly all are written for search engines, not agents. - AI crawler rules — Add explicit allow and deny directives for named AI user agents inside robots.txt. Scored separately from merely having the file.
- XML sitemap — Publish
/sitemap.xmland point to it from robots.txt with aSitemap:directive. Agents treat it as the crawl frontier. - Markdown content negotiation — Answer
Accept: text/markdownwith markdown. Cloudflare reports up to 80% fewer tokens; only 3.9% of top domains do it. - llms.txt — Publish a plain-text reading list of your key pages. Segment it per directory instead of shipping one enormous file.
Lay the technical groundwork
- Content Signals — Declare in robots.txt how crawled content may be used, separating search, AI input, and AI training. Only 4% of top domains declare anything.
- Link headers — Emit discovery relations in HTTP
Linkheaders per RFC 8288, so an agent finds your capabilities without parsing HTML first. - API Catalog — Serve one endpoint per RFC 9727 listing every public API with links to specs and docs. Fewer than 15 of the top domains do.
- DNS-AID — Publish agent-interface discovery records in DNS so an agent can locate your surfaces before it fetches a single page.
- Markdown twins — Serve an
/index.mdequivalent for every HTML page. Cloudflare does this with rewrite Rules rather than duplicating files.
Wire up agent capabilities
- MCP Server Card — Publish a discoverable card describing your MCP server so agents connect on their own instead of waiting for a human to wire it.
- WebMCP — Expose your site's own tools to general-purpose browser agents so they can act in the page rather than scrape it.
- Agent Skills discovery — Publish the skills your site offers so an agent can find them instead of inferring them from your interface.
- OAuth server discovery — Point agents at your authorization server per RFC 9728 so access is properly delegated instead of credential-shared.
- Web Bot Auth — Support the IETF draft that lets bots cryptographically sign requests against published keys, so you can admit good agents by identity.
Open agentic commerce
- Checked but unscored — x402, UCP, ACP, and AP2 are all evaluated by the scanner and none of them currently move your score.
- x402 — Uses HTTP 402 Payment Required so an agent can pay for access inline; Cloudflare pairs it with Web Bot Auth in pay-per-crawl.
Check your work
- Free external scan — isitagentready.com scans any public domain with no account, which means it also works on your competitors.
- Dashboard diagnostics — Cloudflare dashboard, AEO, then Agent Readiness gives per-check pass/fail with evidence and a Copy Agent Prompt.
- Programmatic check — Pass
agentReadinessto the Cloudflare URL Scanner API to retrieve the full check set for any URL. - Benchmark — 📝Cloudflare Radar republishes the same checks weekly across the top 200,000 domains, filterable by category.
Gotchas
- Having robots.txt is not passing — 78% of top domains ship the file; the AI-rules and Content Signals checks are separate and mostly fail.
- Commerce is points that don't exist — implementing x402, UCP, ACP, or AP2 will not raise your score today. Do it for the capability, not the number.
- No published weighting — neither the scanner nor the dashboard discloses category weights or a point formula. Do not reverse-engineer a score target.
- Directory listings are noise — Cloudflare deleted roughly 450 of its own auto-generated listing pages because they gave agents nothing and cost tokens.
- Site type changes the result — the Customize scan filter (All, Content Site, API/Application) changes which checks apply, so two scans can legitimately differ.