Every month Desvela crawls the top 100,000 domains for the four surfaces a visiting AI agent can actually read: ai-catalog.json (Google's ARD standard), llms.txt, agents.md, and the AI-bot directives in robots.txt. This is what we found.
GPTBot block it entirely| Tranco tier | Checked | ai-catalog.json | llms.txt | agents.md |
|---|---|---|---|---|
| top 1K | 1,000 | 2 (0.2%) | 67 (6.7%) | 2 (0.2%) |
| 1K–10K | 9,000 | 18 (0.2%) | 526 (5.8%) | 19 (0.2%) |
| 10K–100K | 89,997 | 125 (0.1%) | 5178 (5.8%) | 958 (1.1%) |
Domains that name each bot in robots.txt, and how many of those shut it out completely (Disallow: /). Before your agent touches a domain, this is the etiquette it should know.
| Bot | Named in robots.txt | Fully blocked |
|---|---|---|
GPTBot | 10,474 | 8,395 (80.2%) |
ClaudeBot | 9,291 | 7,472 (80.4%) |
CCBot | 9,085 | 8,145 (89.7%) |
Google-Extended | 8,446 | 6,929 (82.0%) |
Bytespider | 8,230 | 7,667 (93.2%) |
meta-externalagent | 7,422 | 6,551 (88.3%) |
Applebot-Extended | 7,176 | 6,455 (90.0%) |
PerplexityBot | 4,179 | 2,377 (56.9%) |
anthropic-ai | 3,524 | 2,678 (76.0%) |
cohere-ai | 2,763 | 2,248 (81.4%) |
Claude-Web | 2,689 | 2,122 (78.9%) |
ai-catalog.json (valid, with entries): huggingface.co #1,345 · zapier.com #2,955 · opus.pro #21,089 · roboflow.com #32,436 · clickhouse.com #34,553 · datarobot.com #55,866 · neon.com #97,739 · desvela.dev (that's us — full disclosure) #—
llms.txt: cloudflare.com #3 · github.com #30 · wordpress.org #43 · digicert.com #44 · adobe.com #66 · opera.com #93 · samsung.com #94 · wordpress.com #99 · gravatar.com #137 · kaspersky.com #138 · dropbox.com #139 · reg.ru #145
agents.md: checkpoint.com #337 · paloaltonetworks.com #503 · olx.pl #1,119 · goal7.co #1,701 · razorpay.com #3,699 · nanit.com #4,735 · eufylife.com #4,799 · laravel-news.com #5,057 · nazwa.pl #5,064 · guinnessworldrecords.com #5,707 · jbhifi.com.au #6,051 · network-lc.ru #6,312
Plain GET requests (no rendering) against /.well-known/ai-catalog.json, /llms.txt, /agents.md and /robots.txt on each apex domain of the Tranco top-100K, paced at ~4 req/s globally, honoring robots.txt, as DesvelaBot/0.1 (bot policy). HTML responses to text paths count as absent. Every publisher is re-probed with a canary request to a nonexistent path: servers that answer 200 to anything are excluded as catch-alls, including some seemingly legitimate publishers. Undercounting beats inflating.