How we count.
And what we don't claim.

The census is a measurement of adoption and access — which domains publish files an AI agent can read, and which bot directives they name or block. It is deliberately not a measurement of citations, recommendations or visibility inside AI systems. This page is the single reference for how every edition was collected.

What we look at.

Each edition crawls the top 100,000 domains of the Tranco list, taken on the day of the crawl. Four surfaces per apex domain:

• /.well-known/ai-catalog.json — Google's Agent Readiness Dataset (ARD) standard, spec published May 2026.
• /llms.txt — the LLM-friendly site index.
• /agents.md — agent instructions / capabilities file.
• /robots.txt — the AI-bot directives (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, …) named or blocked with Disallow: /.

Subdomains are out of scope by design: docs.example.com publishing an llms.txt does not count for example.com. The census measures apex-domain adoption, the same bar for everyone.

How we crawl.

Plain GET requests, no rendering, against each path on each apex domain, paced at ~4 requests/second globally, honoring robots.txt, identified as DesvelaBot/0.1 (bot policy).

• HTML responses to text paths count as absent. A server that answers any path with the same HTML page is not publishing a machine-readable file.
• Every publisher is re-probed with a canary request to a nonexistent path. Servers that answer 200 to anything are excluded as catch-alls, including some seemingly legitimate publishers. Undercounting beats inflating.
• The DNS-AID discovery path (_index._agents.<domain> SVCB record, added to the spec 2026-07-02) is probed only on the monthly pass. A positive is discarded unless a wildcard canary for the same zone also fails.

When we publish a number.

Only editions with complete or explicitly declared coverage are published. A crawl that does not finish is not silently reported as full: the generator refuses to emit a census unless at least 95% of the universe was crawled within the last 3 days, and a partial pass must declare its own denominator. An incomplete internal pass that never became a public edition does not appear in the archive, in downloads or in the editions list.

Each edition states, in its manifest, the number of domains checked, the universe size, when the data was collected, when it was published and when (if ever) it was corrected.

What these numbers do not claim.

• Adoption or access signals do not demonstrate citations, recommendations or visibility inside any AI system.
• A domain publishing a file is evidence that the file exists, not that an agent reads it or that it influences an answer.
• Our own domains (desvela.dev, desvela.ai) publish valid manifests and are deliberately absent from every figure on the census pages. They are not in the Tranco top-100K, so counting them would pad the numbers being reported. Same rule for every other domain indexed outside the list.

Where everything lives.

Each edition keeps its own URL, so a number cited last month is still there to check. Every published edition offers a downloadable JSON manifest (provenance + metrics) and a CSV of the tier aggregates.

Latest edition →  ·  July 2026  ·  August 2026