Every month Desvela crawls the top 100,000 domains for the four surfaces a visiting AI agent can actually read: ai-catalog.json (Google's ARD standard), llms.txt, agents.md, and the AI-bot directives in robots.txt. This is what we found.
GPTBot block it entirely| Tranco tier | Checked | ai-catalog.json | llms.txt | agents.md |
|---|---|---|---|---|
| top 1K | 1,000 | 2 (0.2%) | 67 (6.7%) | 2 (0.2%) |
| 1K–10K | 9,000 | 18 (0.2%) | 526 (5.8%) | 19 (0.2%) |
| 10K–100K | 89,997 | 125 (0.1%) | 5178 (5.8%) | 958 (1.1%) |
Domains that name each bot in robots.txt, and how many of those shut it out completely (Disallow: /). Before your agent touches a domain, this is the etiquette it should know.
| Bot | Named in robots.txt | Fully blocked |
|---|---|---|
GPTBot | 10,474 | 8,395 (80.2%) |
ClaudeBot | 9,291 | 7,472 (80.4%) |
CCBot | 9,085 | 8,145 (89.7%) |
Google-Extended | 8,446 | 6,929 (82.0%) |
Bytespider | 8,230 | 7,667 (93.2%) |
meta-externalagent | 7,422 | 6,551 (88.3%) |
Applebot-Extended | 7,176 | 6,455 (90.0%) |
PerplexityBot | 4,179 | 2,377 (56.9%) |
anthropic-ai | 3,524 | 2,678 (76.0%) |
cohere-ai | 2,763 | 2,248 (81.4%) |
Claude-Web | 2,689 | 2,122 (78.9%) |
ai-catalog.json (valid, with entries): huggingface.co #1,345 · zapier.com #2,955 · opus.pro #21,089 · roboflow.com #32,436 · clickhouse.com #34,553 · datarobot.com #55,866 · neon.com #97,739 · desvela.dev (that's us — full disclosure) #—
llms.txt: cloudflare.com #3 · github.com #30 · wordpress.org #43 · digicert.com #44 · adobe.com #66 · opera.com #93 · samsung.com #94 · wordpress.com #99 · gravatar.com #137 · kaspersky.com #138 · dropbox.com #139 · reg.ru #145
agents.md: checkpoint.com #337 · paloaltonetworks.com #503 · olx.pl #1,119 · goal7.co #1,701 · razorpay.com #3,699 · nanit.com #4,735 · eufylife.com #4,799 · laravel-news.com #5,057 · nazwa.pl #5,064 · guinnessworldrecords.com #5,707 · jbhifi.com.au #6,051 · network-lc.ru #6,312
Same four files, same rules, same canary request. You get a grade against the numbers above, and the specific file you are missing. Free, no account, about ten seconds.
Every number on this page is anchored to one crawl, one universe and one method version. These signals measure adoption and access — a domain publishing a file, a bot being named or blocked — and they do not demonstrate citations, recommendations or visibility inside any AI system.
| Field | Value |
|---|---|
| Edition | 2026-07 |
| Data collected | 2026-07-09 |
| Published | 2026-07-10 |
| Corrected | 2026-08-04 |
| Methodology | v1.0 |
| Coverage | 99,997 of 100,000 domains checked |
| Universe | Tranco top-100000 (list date not preserved) |
| Own domains in scope | not preserved for this edition |
JSON (manifest + metrics) · CSV (by tier)
Methodology, in full: how we count.
Each edition stays at its own URL after it is replaced, so a number you cited last month is still there to check. Only editions with complete or explicitly declared coverage are published.
August 2026 · July 2026 (this one)
Plain GET requests (no rendering) against /.well-known/ai-catalog.json, /llms.txt, /agents.md and /robots.txt on each apex domain of the Tranco top-100K, paced at ~4 req/s globally, honoring robots.txt, as DesvelaBot/0.1 (bot policy). HTML responses to text paths count as absent. Every publisher is re-probed with a canary request to a nonexistent path: servers that answer 200 to anything are excluded as catch-alls, including some seemingly legitimate publishers. Undercounting beats inflating.
Read the full methodology → (universe, exclusions, coverage rules and what these numbers do not claim).
One email per month with the fresh numbers and what changed: new publishers, withdrawals, bot-blocking shifts. No product spam. Unsubscribe in one click from any email. We email you a confirmation link first, and nothing is sent until you click it.