Every month Desvela crawls the top 100,000 domains for the files an AI agent could read. Then it asks each of them for a path that cannot exist. A server that answers that answers anything, so everything it appears to publish is an artefact of the server, not a decision by its owner. This month that removed 75% of the domains that appear to publish Google's ARD manifest.
ai-catalog.json — down from 542 that answer on that pathmcp.json — down from 498 that answer on that pathard.json — down from 456 that answer on that pathagent-card.json — down from 435 that answer on that path| Tranco tier | Checked | ai-catalog.json | llms.txt | agents.md | agent-card.json | mcp.json | ard.json |
|---|---|---|---|---|---|---|---|
| top 1K | 1,000 | 2 (0.2%) | 87 (8.7%) | 2 (0.2%) | 1 (0.1%) | 2 (0.2%) | 2 (0.2%) |
| 1K–10K | 9,000 | 17 (0.2%) | 637 (7.1%) | 24 (0.3%) | 8 (0.1%) | 13 (0.1%) | 8 (0.1%) |
| 10K–100K | 89,998 | 107 (0.1%) | 6669 (7.4%) | 1015 (1.1%) | 49 (0.1%) | 98 (0.1%) | 45 (0.1%) |
Domains that name each bot in robots.txt, and how many of those shut it out completely (Disallow: /). Before your agent touches a domain, this is the etiquette it should know.
| Bot | Named in robots.txt | Fully blocked |
|---|---|---|
GPTBot | 10,666 | 8,109 (76.0%) |
ClaudeBot | 9,765 | 7,387 (75.6%) |
CCBot | 9,193 | 7,903 (86.0%) |
Google-Extended | 8,748 | 6,688 (76.5%) |
Bytespider | 8,495 | 7,743 (91.1%) |
meta-externalagent | 7,823 | 6,702 (85.7%) |
Applebot-Extended | 7,595 | 6,519 (85.8%) |
anthropic-ai | 7,433 | 6,430 (86.5%) |
cohere-ai | 6,675 | 6,059 (90.8%) |
Claude-Web | 6,444 | 5,771 (89.6%) |
PerplexityBot | 4,909 | 2,378 (48.4%) |
ChatGPT-User | 4,822 | 2,422 (50.2%) |
OAI-SearchBot | 3,767 | 1,420 (37.7%) |
Perplexity-User | 2,223 | 1,120 (50.4%) |
Claude-User | 2,182 | 1,086 (49.8%) |
meta-externalfetcher | 1,650 | 1,275 (77.3%) |
DuckAssistBot | 1,498 | 996 (66.5%) |
MistralAI-User | 1,272 | 963 (75.7%) |
Google-NotebookLM | 588 | 513 (87.2%) |
Gemini-Deep-Research | 494 | 358 (72.5%) |
Google-Agent | 468 | 410 (87.6%) |
NovaAct | 373 | 342 (91.7%) |
Devin | 363 | 345 (95.0%) |
Kimi-User | 320 | 314 (98.1%) |
GoogleAgent-Mariner | 315 | 275 (87.3%) |
AmazonBuyForMe | 280 | 271 (96.8%) |
Manus-User | 263 | 247 (93.9%) |
TwinAgent | 229 | 224 (97.8%) |
Kagi-Fetcher | 109 | 104 (95.4%) |
Claude-Code | 98 | 78 (79.6%) |
OpenCode | 65 | 62 (95.4%) |
Trae | 64 | 62 (96.9%) |
GoogleAgent-URLContext | 63 | 55 (87.3%) |
Cursor | 58 | 55 (94.8%) |
YepBot | 44 | 44 (100.0%) |
Google-Gemini-CLI | 40 | 36 (90.0%) |
ChatGPT-Agent | 10 | 6 (60.0%) |
DesvelaBot | 1 | 1 (100.0%) |
The table above is what domains declare in robots.txt. This one is what happened when we sent each AI crawler's real user-agent at the homepage of the Tranco top-1,000 and compared the response to a browser's, from the same connection. The headline is not the blocking: 413 of 999 domains (41.3%) never answered our browser-labelled control either — they refuse any plain HTTP client, whoever it claims to be.
| Crawler | Blocked (4xx) | Throttled (429) | Degraded (<60% of bytes) | Did not get the page |
|---|---|---|---|---|
ClaudeBot | 74 | 12 | 8 | 16.0% |
GPTBot | 73 | 4 | 9 | 14.7% |
CCBot | 69 | 7 | 9 | 14.5% |
PerplexityBot | 63 | 3 | 11 | 13.1% |
OAI-SearchBot | 52 | 3 | 8 | 10.8% |
Rates are over the 586 measurable domains — the ones that answered a plain client at all, which are by definition the most permissive of the cohort. The true rate across the full 1,000 is higher and cannot be measured this way. A 429 is counted apart from a block because it says "too fast", not "not you". Swept 2026-09-01 from a residential connection; the vantage is part of the method — datacenter IPs get refused far more often.
The ARD spec added a DNS path in July 2026: publish an SVCB record at _index._agents.<domain> and an agent can find your registry without fetching anything over HTTP. We probed 98,756 domains and 42 publish one, which is 0.043% or about 1 in 2,351.
Sector is the content supercategory Cloudflare Intel assigns to each domain, recorded with its source and date — we classify nothing by hand. 99,998 of the 99,998 domains in scope are classified, 93,167 with at least one category. A domain with two supercategories counts in both, so the rows do not add up to the total.
| Sector | Domains | llms.txt | agents.md | Manifest (≥1 entry) | Names an AI bot | Blocks GPTBot |
|---|---|---|---|---|---|---|
| Technology | 29,665 | 2,520 (8.5%) | 150 (0.5%) | 74 (0.2%) | 2,218 (7.5%) | 1,068 (3.6%) |
| Entertainment | 16,698 | 938 (5.6%) | 45 (0.3%) | 6 (0.0%) | 4,365 (26.1%) | 3,270 (19.6%) |
| Business & Economy | 14,496 | 1,995 (13.8%) | 119 (0.8%) | 55 (0.4%) | 1,484 (10.2%) | 608 (4.2%) |
| CIPA | 8,955 | 318 (3.6%) | 13 (0.1%) | 3 (0.0%) | 1,545 (17.3%) | 1,233 (13.8%) |
| Education | 8,074 | 430 (5.3%) | 20 (0.2%) | 8 (0.1%) | 1,135 (14.1%) | 749 (9.3%) |
| Shopping & Auctions | 7,803 | 1,371 (17.6%) | 773 (9.9%) | 14 (0.2%) | 1,369 (17.5%) | 601 (7.7%) |
| Society & Lifestyle | 6,384 | 992 (15.5%) | 558 (8.7%) | 7 (0.1%) | 1,128 (17.7%) | 621 (9.7%) |
| Government & Politics | 5,995 | 233 (3.9%) | 6 (0.1%) | 3 (0.1%) | 454 (7.6%) | 301 (5.0%) |
| Internet Communication | 4,759 | 348 (7.3%) | 8 (0.2%) | 8 (0.2%) | 831 (17.5%) | 575 (12.1%) |
| Adult Themes | 4,065 | 133 (3.3%) | 5 (0.1%) | 2 (0.0%) | 741 (18.2%) | 573 (14.1%) |
| Gambling | 2,754 | 76 (2.8%) | 0 (0.0%) | 1 (0.0%) | 454 (16.5%) | 364 (13.2%) |
| Health | 2,460 | 267 (10.9%) | 41 (1.7%) | 1 (0.0%) | 334 (13.6%) | 177 (7.2%) |
| Travel | 2,340 | 200 (8.5%) | 1 (0.0%) | 9 (0.4%) | 376 (16.1%) | 155 (6.6%) |
| Ads | 2,171 | 253 (11.7%) | 5 (0.2%) | 8 (0.4%) | 113 (5.2%) | 30 (1.4%) |
| Sports | 2,023 | 117 (5.8%) | 22 (1.1%) | 2 (0.1%) | 462 (22.8%) | 259 (12.8%) |
| Miscellaneous | 1,543 | 32 (2.1%) | 3 (0.2%) | 0 (0.0%) | 77 (5.0%) | 66 (4.3%) |
| Vehicles | 1,474 | 124 (8.4%) | 11 (0.7%) | 4 (0.3%) | 241 (16.4%) | 133 (9.0%) |
| Questionable Content | 1,061 | 45 (4.2%) | 3 (0.3%) | 0 (0.0%) | 281 (26.5%) | 261 (24.6%) |
| Job Search & Careers | 859 | 82 (9.5%) | 2 (0.2%) | 2 (0.2%) | 134 (15.6%) | 70 (8.1%) |
| Real Estate | 761 | 69 (9.1%) | 3 (0.4%) | 1 (0.1%) | 105 (13.8%) | 40 (5.3%) |
| Religion | 342 | 12 (3.5%) | 1 (0.3%) | 0 (0.0%) | 62 (18.1%) | 40 (11.7%) |
| Weather | 237 | 16 (6.8%) | 1 (0.4%) | 0 (0.0%) | 39 (16.5%) | 22 (9.3%) |
| Military & Weapons | 202 | 18 (8.9%) | 2 (1.0%) | 0 (0.0%) | 22 (10.9%) | 17 (8.4%) |
| Safe for Kids | 178 | 12 (6.7%) | 10 (5.6%) | 0 (0.0%) | 32 (18.0%) | 22 (12.4%) |
| (not categorized by source) | 6,831 | 138 (2.0%) | 9 (0.1%) | 5 (0.1%) | 394 (5.8%) | 307 (4.5%) |
The sector table above counts every file that is there. Some of them were not written by the site: the platform serves the same template to all its tenants. Here the two are separated, per sector.
| Sector | Domains | llms.txt present | Served as a platform template | Written by the site |
|---|---|---|---|---|
| All domains in scope | 99,998 | 7,393 (7.4%) | 836 (11.3%) | 4,901 (4.9%) |
| Technology | 29,665 | 2,520 (8.5%) | 91 (3.6%) | 1,782 (6.0%) |
| Entertainment | 16,698 | 938 (5.6%) | 37 (3.9%) | 639 (3.8%) |
| Business & Economy | 14,496 | 1,995 (13.8%) | 82 (4.1%) | 1,442 (9.9%) |
| CIPA | 8,955 | 318 (3.6%) | 12 (3.8%) | 206 (2.3%) |
| Education | 8,074 | 430 (5.3%) | 19 (4.4%) | 314 (3.9%) |
| Shopping & Auctions | 7,803 | 1,371 (17.6%) | 653 (47.6%) | 585 (7.5%) |
| Society & Lifestyle | 6,384 | 992 (15.5%) | 488 (49.2%) | 386 (6.0%) |
| Government & Politics | 5,995 | 233 (3.9%) | 2 (0.9%) | 165 (2.8%) |
| Internet Communication | 4,759 | 348 (7.3%) | 6 (1.7%) | 243 (5.1%) |
| Adult Themes | 4,065 | 133 (3.3%) | 5 (3.8%) | 67 (1.6%) |
| Gambling | 2,754 | 76 (2.8%) | 0 (0.0%) | 58 (2.1%) |
| Health | 2,460 | 267 (10.9%) | 29 (10.9%) | 189 (7.7%) |
| Travel | 2,340 | 200 (8.5%) | 1 (0.5%) | 160 (6.8%) |
| Ads | 2,171 | 253 (11.7%) | 3 (1.2%) | 177 (8.2%) |
| Sports | 2,023 | 117 (5.8%) | 20 (17.1%) | 71 (3.5%) |
| Miscellaneous | 1,543 | 32 (2.1%) | 3 (9.4%) | 25 (1.6%) |
| Vehicles | 1,474 | 124 (8.4%) | 10 (8.1%) | 80 (5.4%) |
| Questionable Content | 1,061 | 45 (4.2%) | 3 (6.7%) | 30 (2.8%) |
| Job Search & Careers | 859 | 82 (9.5%) | 1 (1.2%) | 59 (6.9%) |
| Real Estate | 761 | 69 (9.1%) | 3 (4.3%) | 59 (7.8%) |
| Religion | 342 | 12 (3.5%) | 1 (8.3%) | 11 (3.2%) |
| Weather | 237 | 16 (6.8%) | 0 (0.0%) | 14 (5.9%) |
| Military & Weapons | 202 | 18 (8.9%) | 2 (11.1%) | 13 (6.4%) |
| Safe for Kids | 178 | 12 (6.7%) | 7 (58.3%) | 5 (2.8%) |
| (not categorized by source) | 6,831 | 138 (2.0%) | 5 (3.6%) | 99 (1.4%) |
The source's own categories, with the sector each one sits under. Where a sector hides a contrast, the category shows it. Every figure is a share of the domains in that category.
| Category (sector) | Domains | llms.txt | Written by the site | Platform template | agents.md | Names an AI bot | Blocks GPTBot |
|---|---|---|---|---|---|---|---|
| Technology (Technology) | 11,769 | 13.9% | 9.7% | 0.6% | 1.0% | 10.9% | 4.9% |
| Business (Business & Economy) | 10,639 | 14.9% | 10.6% | 0.8% | 1.0% | 10.2% | 3.9% |
| Content Servers (Technology) | 9,724 | 0.6% | 0.4% | 0.1% | 0.1% | 1.6% | 1.2% |
| CIPA Filter (CIPA) | 8,955 | 3.6% | 2.3% | 0.1% | 0.1% | 17.3% | 13.8% |
| News & Media (Entertainment) | 6,683 | 6.2% | 4.3% | 0.0% | 0.1% | 33.0% | 25.4% |
| Ecommerce (Shopping & Auctions) | 6,150 | 19.8% | 7.8% | 10.4% | 12.3% | 15.4% | 5.9% |
| Information Technology (Technology) | 5,041 | 9.3% | 6.7% | 0.2% | 0.4% | 6.0% | 2.6% |
| Politics, Advocacy, and Government-Related (Government & Politics) | 4,568 | 3.1% | 1.9% | 0.0% | 0.0% | 6.7% | 4.5% |
| Education (Education) | 3,927 | 6.1% | 4.2% | 0.5% | 0.5% | 16.2% | 10.8% |
| Pornography (Adult Themes) | 3,689 | 2.9% | 1.5% | 0.1% | 0.1% | 18.6% | 14.5% |
| Gaming (Entertainment) | 3,422 | 4.4% | 3.0% | 0.3% | 0.2% | 16.1% | 12.9% |
| Video Streaming (Entertainment) | 3,379 | 4.5% | 3.1% | 0.1% | 0.1% | 32.7% | 25.9% |
| Educational Institutions (Education) | 3,308 | 4.4% | 3.4% | 0.0% | 0.0% | 7.9% | 4.4% |
| Gambling (Gambling) | 2,754 | 2.8% | 2.1% | 0.0% | 0.0% | 16.5% | 13.2% |
| Health & Fitness (Health) | 2,451 | 10.9% | 7.7% | 1.2% | 1.7% | 13.6% | 7.2% |
| Travel (Travel) | 2,340 | 8.5% | 6.8% | 0.0% | 0.0% | 16.1% | 6.6% |
| Sports (Sports) | 2,023 | 5.8% | 3.5% | 1.0% | 1.1% | 22.8% | 12.8% |
| Personal Finance (Business & Economy) | 1,916 | 9.8% | 7.3% | 0.0% | 0.2% | 7.2% | 2.6% |
| Trackers/Analytics (Ads) | 1,897 | 12.7% | 8.9% | 0.1% | 0.3% | 5.2% | 1.0% |
| Economy & Finance (Business & Economy) | 1,710 | 10.5% | 8.3% | 0.1% | 0.1% | 12.7% | 6.8% |
| Shareware/Freeware (Technology) | 1,676 | 11.0% | 8.8% | 0.4% | 0.5% | 13.2% | 6.9% |
| Personal Blogs (Internet Communication) | 1,648 | 7.0% | 4.4% | 0.2% | 0.2% | 17.2% | 12.0% |
| Vehicles (Vehicles) | 1,474 | 8.4% | 5.4% | 0.7% | 0.7% | 16.4% | 9.0% |
| Fashion (Society & Lifestyle) | 1,360 | 28.9% | 8.4% | 19.0% | 21.5% | 15.8% | 6.0% |
| Clothing (Society & Lifestyle) | 1,281 | 28.6% | 6.2% | 20.0% | 22.1% | 14.3% | 6.6% |
| Entertainment (Entertainment) | 1,193 | 5.9% | 4.4% | 0.0% | 0.3% | 21.6% | 12.9% |
| Search Engines (Technology) | 1,173 | 5.4% | 4.1% | 0.1% | 0.3% | 16.2% | 8.9% |
| Movies (Entertainment) | 1,169 | 3.7% | 2.7% | 0.1% | 0.2% | 25.1% | 21.4% |
| Login Screens (Miscellaneous) | 1,155 | 2.1% | 1.5% | 0.3% | 0.3% | 6.0% | 5.2% |
| Forums (Internet Communication) | 1,149 | 3.7% | 2.8% | 0.0% | 0.0% | 28.8% | 24.9% |
On 11 October 2026 the ARD spec (v0.91, §3.3) listed the recognized media types for agentic resources. This is the first edition since, so it is the baseline. Every indexed entry's type is sorted into a §3.3 family: the exact §3.3 name counts as recognized, any other spelling of the same kind of resource as a variant. A domain using both counts in both columns.
| Kind of resource | Domains | Using the §3.3 name | Using a variant | Variants seen |
|---|---|---|---|---|
| MCP server | 112 | 108 (96.4%) | 6 | application/mcp-server+json, application/vnd.mcp+json |
| Agent card | 4 | 4 (100.0%) | 0 | none |
| Agent skill | 42 | 23 (54.8%) | 20 | application/ai-skill, application/ai-skill+md, application/ai-skill-archive+gzip, application/agent-skills-index+json +2 more |
| Agent plugin | 1 | 0 (0.0%) | 1 | application/agent-plugins+json, application/vnd.anthropic.claude-plugin+json, application/vnd.cursor.cursor-plugin+json, application/vnd.github.copilot-plugin +4 more |
| Registry | 2 | 2 (100.0%) | 0 | none |
| Nested catalog | 3 | 3 (100.0%) | 0 | none |
ai-catalog.json (valid, with entries): github.com #34 · shop.app #545 · huggingface.co #1,079 · oxylabs.io #1,333 · hostinger.com #1,832 · bitdefender.com #1,844 · padlet.com #2,720 · cookieyes.com #2,749 · rudderstack.com #2,854 · zapier.com #3,005 · clerk.com #3,430 · hootsuite.com #3,449
llms.txt: cloudflare.com #2 · github.com #34 · adobe.com #61 · wordpress.org #65 · opera.com #76 · wordpress.com #99 · intuit.com #114 · samsung.com #122 · dropbox.com #136 · webex.com #146 · gravatar.com #167 · ubuntu.com #168
agents.md: shopify.com #172 · lg.com #916 · coinbase.com #1,477 · cookieyes.com #2,749 · crazyegg.com #2,988 · backblaze.com #3,334 · razorpay.com #3,369 · nazwa.pl #3,833 · nanit.com #3,942 · eufylife.com #4,848 · nownodes.io #5,585 · laravel-news.com #5,780
Same four files, same rules, same canary request. You get a grade against the numbers above, and the specific file you are missing. Free, no account, about ten seconds.
Every number on this page is anchored to one crawl, one universe and one method version. These signals measure adoption and access — a domain publishing a file, a bot being named or blocked — and they do not demonstrate citations, recommendations or visibility inside any AI system.
| Field | Value |
|---|---|
| Edition | 2026-11 |
| Data collected | 2026-11-02 |
| Published | 2026-11-10 |
| Corrected | — |
| Methodology | v2.0 |
| Coverage | 99,998 of 100,000 domains checked |
| Universe | Tranco top-100000 (list 2026-11-01) |
| Own domains in scope | none |
JSON (manifest + metrics) · CSV (by tier)
Methodology, in full: how we count.
Each edition stays at its own URL after it is replaced, so a number you cited last month is still there to check. Only editions with complete or explicitly declared coverage are published.
November 2026 (this one) · September 2026 · August 2026 · July 2026
Plain GET requests (no rendering) on each apex domain of the Tranco top-100K: /robots.txt first, then whichever of these six paths it allows: /.well-known/ai-catalog.json, /.well-known/ard.json, /.well-known/agent-card.json, /.well-known/mcp.json, /llms.txt and /agents.md. Paced at ~4 req/s globally, as DesvelaBot/0.1 (bot policy). HTML responses to text paths count as absent. Every publisher is re-probed with two canary requests to paths that cannot exist, one at the root and one under /.well-known/: servers that answer 2xx to either are excluded as catch-alls, including some seemingly legitimate publishers. Undercounting beats inflating. Separately, the MCP server cards that indexed catalogs declare are fetched to check they validate, and the DNS-AID record is looked up through a public resolver, which never touches the domain's own server.
Read the full methodology → (universe, exclusions, coverage rules and what these numbers do not claim).
One email per month with the fresh numbers and what changed: new publishers, withdrawals, bot-blocking shifts. No product spam. Unsubscribe in one click from any email. We email you a confirmation link first, and nothing is sent until you click it.