State of the agent-readable web · November 2026

Most of what looks like
an agent-readable web is not there.

Every month Desvela crawls the top 100,000 domains for the files an AI agent could read. Then it asks each of them for a path that cannot exist. A server that answers that answers anything, so everything it appears to publish is an artefact of the server, not a decision by its owner. This month that removed 75% of the domains that appear to publish Google's ARD manifest.

99,998
domains checked this pass
137
real ai-catalog.json — down from 542 that answer on that path
113
real mcp.json — down from 498 that answer on that path
65
real ard.json — down from 456 that answer on that path
58
real agent-card.json — down from 435 that answer on that path

See where your own domain lands →

Adoption

The four surfaces we have measured longest, by ranking tier.

Tranco tierCheckedai-catalog.jsonllms.txtagents.mdagent-card.jsonmcp.jsonard.json
top 1K1,000 2 (0.2%) 87 (8.7%) 2 (0.2%) 1 (0.1%) 2 (0.2%) 2 (0.2%)
1K–10K9,000 17 (0.2%) 637 (7.1%) 24 (0.3%) 8 (0.1%) 13 (0.1%) 8 (0.1%)
10K–100K89,998 107 (0.1%) 6669 (7.4%) 1015 (1.1%) 49 (0.1%) 98 (0.1%) 45 (0.1%)
ai-catalog.json reality check: 137 files respond — 3 are invalid JSON, and only 126 are valid manifests with at least one entry: github.com, shop.app, huggingface.co, oxylabs.io, hostinger.com, bitdefender.com, padlet.com, cookieyes.com, rudderstack.com, zapier.com, clerk.com, hootsuite.com, netlify.com, supabase.com, beehiiv.com. Google published the ARD spec in May 2026; adoption is a rounding error.
The door policy

Who's blocking the AI bots.

Domains that name each bot in robots.txt, and how many of those shut it out completely (Disallow: /). Before your agent touches a domain, this is the etiquette it should know.

BotNamed in robots.txtFully blocked
GPTBot10,666 8,109 (76.0%)
ClaudeBot9,765 7,387 (75.6%)
CCBot9,193 7,903 (86.0%)
Google-Extended8,748 6,688 (76.5%)
Bytespider8,495 7,743 (91.1%)
meta-externalagent7,823 6,702 (85.7%)
Applebot-Extended7,595 6,519 (85.8%)
anthropic-ai7,433 6,430 (86.5%)
cohere-ai6,675 6,059 (90.8%)
Claude-Web6,444 5,771 (89.6%)
PerplexityBot4,909 2,378 (48.4%)
ChatGPT-User4,822 2,422 (50.2%)
OAI-SearchBot3,767 1,420 (37.7%)
Perplexity-User2,223 1,120 (50.4%)
Claude-User2,182 1,086 (49.8%)
meta-externalfetcher1,650 1,275 (77.3%)
DuckAssistBot1,498 996 (66.5%)
MistralAI-User1,272 963 (75.7%)
Google-NotebookLM588 513 (87.2%)
Gemini-Deep-Research494 358 (72.5%)
Google-Agent468 410 (87.6%)
NovaAct373 342 (91.7%)
Devin363 345 (95.0%)
Kimi-User320 314 (98.1%)
GoogleAgent-Mariner315 275 (87.3%)
AmazonBuyForMe280 271 (96.8%)
Manus-User263 247 (93.9%)
TwinAgent229 224 (97.8%)
Kagi-Fetcher109 104 (95.4%)
Claude-Code98 78 (79.6%)
OpenCode65 62 (95.4%)
Trae64 62 (96.9%)
GoogleAgent-URLContext63 55 (87.3%)
Cursor58 55 (94.8%)
YepBot44 44 (100.0%)
Google-Gemini-CLI40 36 (90.0%)
ChatGPT-Agent10 6 (60.0%)
DesvelaBot1 1 (100.0%)
The door policy, observed

What the servers actually do when the crawler shows up.

The table above is what domains declare in robots.txt. This one is what happened when we sent each AI crawler's real user-agent at the homepage of the Tranco top-1,000 and compared the response to a browser's, from the same connection. The headline is not the blocking: 413 of 999 domains (41.3%) never answered our browser-labelled control either — they refuse any plain HTTP client, whoever it claims to be.

CrawlerBlocked (4xx)Throttled (429)Degraded (<60% of bytes)Did not get the page
ClaudeBot74128 16.0%
GPTBot7349 14.7%
CCBot6979 14.5%
PerplexityBot63311 13.1%
OAI-SearchBot5238 10.8%

Rates are over the 586 measurable domains — the ones that answered a plain client at all, which are by definition the most permissive of the cohort. The true rate across the full 1,000 is higher and cannot be measured this way. A 429 is counted apart from a block because it says "too fast", not "not you". Swept 2026-09-01 from a residential connection; the vantage is part of the method — datacenter IPs get refused far more often.

Third-party context, not our measurement: ChatGPT holds 53.9% of assistant web visits and Gemini 27.9% (Similarweb, May 2026, sessions on each assistant's own domain) — yet by referral share arriving at websites, Statcounter (August 2026, its ~1.5M-site tracking network) puts ChatGPT at 79.4% and Gemini at 10.9%, with Perplexity at 4.3% on a 1.3% visit share. The two sources measure different questions and disagree 3× — quote either alone and you have picked a side without saying so. Read together with the table above: the crawler treated best (OAI-SearchBot) belongs to the assistant that sends the most traffic back.
DNS discovery

Almost nobody publishes the DNS record.

The ARD spec added a DNS path in July 2026: publish an SVCB record at _index._agents.<domain> and an agent can find your registry without fetching anything over HTTP. We probed 98,756 domains and 42 publish one, which is 0.043% or about 1 in 2,351.

Every positive is checked against a wildcard canary: we ask the same zone for a name nobody could have published, and if that answers too the positive is discarded. Without that check the number would be higher and wrong.
By sector

Which sectors have set the table, and which shut the door.

Sector is the content supercategory Cloudflare Intel assigns to each domain, recorded with its source and date — we classify nothing by hand. 99,998 of the 99,998 domains in scope are classified, 93,167 with at least one category. A domain with two supercategories counts in both, so the rows do not add up to the total.

SectorDomainsllms.txtagents.mdManifest (≥1 entry)Names an AI botBlocks GPTBot
Technology29,6652,520 (8.5%)150 (0.5%)74 (0.2%)2,218 (7.5%)1,068 (3.6%)
Entertainment16,698938 (5.6%)45 (0.3%)6 (0.0%)4,365 (26.1%)3,270 (19.6%)
Business & Economy14,4961,995 (13.8%)119 (0.8%)55 (0.4%)1,484 (10.2%)608 (4.2%)
CIPA8,955318 (3.6%)13 (0.1%)3 (0.0%)1,545 (17.3%)1,233 (13.8%)
Education8,074430 (5.3%)20 (0.2%)8 (0.1%)1,135 (14.1%)749 (9.3%)
Shopping & Auctions7,8031,371 (17.6%)773 (9.9%)14 (0.2%)1,369 (17.5%)601 (7.7%)
Society & Lifestyle6,384992 (15.5%)558 (8.7%)7 (0.1%)1,128 (17.7%)621 (9.7%)
Government & Politics5,995233 (3.9%)6 (0.1%)3 (0.1%)454 (7.6%)301 (5.0%)
Internet Communication4,759348 (7.3%)8 (0.2%)8 (0.2%)831 (17.5%)575 (12.1%)
Adult Themes4,065133 (3.3%)5 (0.1%)2 (0.0%)741 (18.2%)573 (14.1%)
Gambling2,75476 (2.8%)0 (0.0%)1 (0.0%)454 (16.5%)364 (13.2%)
Health2,460267 (10.9%)41 (1.7%)1 (0.0%)334 (13.6%)177 (7.2%)
Travel2,340200 (8.5%)1 (0.0%)9 (0.4%)376 (16.1%)155 (6.6%)
Ads2,171253 (11.7%)5 (0.2%)8 (0.4%)113 (5.2%)30 (1.4%)
Sports2,023117 (5.8%)22 (1.1%)2 (0.1%)462 (22.8%)259 (12.8%)
Miscellaneous1,54332 (2.1%)3 (0.2%)0 (0.0%)77 (5.0%)66 (4.3%)
Vehicles1,474124 (8.4%)11 (0.7%)4 (0.3%)241 (16.4%)133 (9.0%)
Questionable Content1,06145 (4.2%)3 (0.3%)0 (0.0%)281 (26.5%)261 (24.6%)
Job Search & Careers85982 (9.5%)2 (0.2%)2 (0.2%)134 (15.6%)70 (8.1%)
Real Estate76169 (9.1%)3 (0.4%)1 (0.1%)105 (13.8%)40 (5.3%)
Religion34212 (3.5%)1 (0.3%)0 (0.0%)62 (18.1%)40 (11.7%)
Weather23716 (6.8%)1 (0.4%)0 (0.0%)39 (16.5%)22 (9.3%)
Military & Weapons20218 (8.9%)2 (1.0%)0 (0.0%)22 (10.9%)17 (8.4%)
Safe for Kids17812 (6.7%)10 (5.6%)0 (0.0%)32 (18.0%)22 (12.4%)
(not categorized by source)6,831138 (2.0%)9 (0.1%)5 (0.1%)394 (5.8%)307 (4.5%)
2 sectors with fewer than 100 domains are omitted from the table; the full breakdown is in the downloadable JSON. Categories such as CIPA or Ads are content filters, not industries: read them as what the source says, not as a market.
By sector, by author

llms.txt: written by the site, or served by its platform.

The sector table above counts every file that is there. Some of them were not written by the site: the platform serves the same template to all its tenants. Here the two are separated, per sector.

SectorDomainsllms.txt presentServed as a platform templateWritten by the site
All domains in scope99,9987,393 (7.4%)836 (11.3%)4,901 (4.9%)
Technology29,6652,520 (8.5%)91 (3.6%)1,782 (6.0%)
Entertainment16,698938 (5.6%)37 (3.9%)639 (3.8%)
Business & Economy14,4961,995 (13.8%)82 (4.1%)1,442 (9.9%)
CIPA8,955318 (3.6%)12 (3.8%)206 (2.3%)
Education8,074430 (5.3%)19 (4.4%)314 (3.9%)
Shopping & Auctions7,8031,371 (17.6%)653 (47.6%)585 (7.5%)
Society & Lifestyle6,384992 (15.5%)488 (49.2%)386 (6.0%)
Government & Politics5,995233 (3.9%)2 (0.9%)165 (2.8%)
Internet Communication4,759348 (7.3%)6 (1.7%)243 (5.1%)
Adult Themes4,065133 (3.3%)5 (3.8%)67 (1.6%)
Gambling2,75476 (2.8%)0 (0.0%)58 (2.1%)
Health2,460267 (10.9%)29 (10.9%)189 (7.7%)
Travel2,340200 (8.5%)1 (0.5%)160 (6.8%)
Ads2,171253 (11.7%)3 (1.2%)177 (8.2%)
Sports2,023117 (5.8%)20 (17.1%)71 (3.5%)
Miscellaneous1,54332 (2.1%)3 (9.4%)25 (1.6%)
Vehicles1,474124 (8.4%)10 (8.1%)80 (5.4%)
Questionable Content1,06145 (4.2%)3 (6.7%)30 (2.8%)
Job Search & Careers85982 (9.5%)1 (1.2%)59 (6.9%)
Real Estate76169 (9.1%)3 (4.3%)59 (7.8%)
Religion34212 (3.5%)1 (8.3%)11 (3.2%)
Weather23716 (6.8%)0 (0.0%)14 (5.9%)
Military & Weapons20218 (8.9%)2 (11.1%)13 (6.4%)
Safe for Kids17812 (6.7%)7 (58.3%)5 (2.8%)
(not categorized by source)6,831138 (2.0%)5 (3.6%)99 (1.4%)
Platform template: the file's fingerprint (title, URLs and domain tokens stripped out) is shared by 10 or more domains of the list, counted over the whole list rather than per sector, and shown as a share of the files present. Written by the site: fingerprinted and not a platform template, as a share of all domains in the sector. 1,656 files in this edition have no fingerprint and count as neither, so the written-by-the-site figures are a floor. The same split for agents.md: 1,041 domains publish one, 879 of them (84.4%) serve a platform template and 145 wrote their own. A sector with fewer than 100 domains is left out of the table; the full breakdown is in the downloadable JSON.
Finer grain

The same table, one level down.

The source's own categories, with the sector each one sits under. Where a sector hides a contrast, the category shows it. Every figure is a share of the domains in that category.

Category (sector)Domainsllms.txtWritten by the sitePlatform templateagents.mdNames an AI botBlocks GPTBot
Technology (Technology)11,76913.9%9.7%0.6%1.0%10.9%4.9%
Business (Business & Economy)10,63914.9%10.6%0.8%1.0%10.2%3.9%
Content Servers (Technology)9,7240.6%0.4%0.1%0.1%1.6%1.2%
CIPA Filter (CIPA)8,9553.6%2.3%0.1%0.1%17.3%13.8%
News & Media (Entertainment)6,6836.2%4.3%0.0%0.1%33.0%25.4%
Ecommerce (Shopping & Auctions)6,15019.8%7.8%10.4%12.3%15.4%5.9%
Information Technology (Technology)5,0419.3%6.7%0.2%0.4%6.0%2.6%
Politics, Advocacy, and Government-Related (Government & Politics)4,5683.1%1.9%0.0%0.0%6.7%4.5%
Education (Education)3,9276.1%4.2%0.5%0.5%16.2%10.8%
Pornography (Adult Themes)3,6892.9%1.5%0.1%0.1%18.6%14.5%
Gaming (Entertainment)3,4224.4%3.0%0.3%0.2%16.1%12.9%
Video Streaming (Entertainment)3,3794.5%3.1%0.1%0.1%32.7%25.9%
Educational Institutions (Education)3,3084.4%3.4%0.0%0.0%7.9%4.4%
Gambling (Gambling)2,7542.8%2.1%0.0%0.0%16.5%13.2%
Health & Fitness (Health)2,45110.9%7.7%1.2%1.7%13.6%7.2%
Travel (Travel)2,3408.5%6.8%0.0%0.0%16.1%6.6%
Sports (Sports)2,0235.8%3.5%1.0%1.1%22.8%12.8%
Personal Finance (Business & Economy)1,9169.8%7.3%0.0%0.2%7.2%2.6%
Trackers/Analytics (Ads)1,89712.7%8.9%0.1%0.3%5.2%1.0%
Economy & Finance (Business & Economy)1,71010.5%8.3%0.1%0.1%12.7%6.8%
Shareware/Freeware (Technology)1,67611.0%8.8%0.4%0.5%13.2%6.9%
Personal Blogs (Internet Communication)1,6487.0%4.4%0.2%0.2%17.2%12.0%
Vehicles (Vehicles)1,4748.4%5.4%0.7%0.7%16.4%9.0%
Fashion (Society & Lifestyle)1,36028.9%8.4%19.0%21.5%15.8%6.0%
Clothing (Society & Lifestyle)1,28128.6%6.2%20.0%22.1%14.3%6.6%
Entertainment (Entertainment)1,1935.9%4.4%0.0%0.3%21.6%12.9%
Search Engines (Technology)1,1735.4%4.1%0.1%0.3%16.2%8.9%
Movies (Entertainment)1,1693.7%2.7%0.1%0.2%25.1%21.4%
Login Screens (Miscellaneous)1,1552.1%1.5%0.3%0.3%6.0%5.2%
Forums (Internet Communication)1,1493.7%2.8%0.0%0.0%28.8%24.9%
Categories with 500+ domains, the 30 largest; 81 categories are in the downloadable JSON only. A domain can sit in several categories and counts in each. Some categories are content filters rather than industries (CIPA Filter, Content Servers, Login Screens): read them as what the source says, not as a market.
Naming

Does the web use the spec's names?

On 11 October 2026 the ARD spec (v0.91, §3.3) listed the recognized media types for agentic resources. This is the first edition since, so it is the baseline. Every indexed entry's type is sorted into a §3.3 family: the exact §3.3 name counts as recognized, any other spelling of the same kind of resource as a variant. A domain using both counts in both columns.

Kind of resourceDomainsUsing the §3.3 nameUsing a variantVariants seen
MCP server112108 (96.4%)6application/mcp-server+json, application/vnd.mcp+json
Agent card44 (100.0%)0none
Agent skill4223 (54.8%)20application/ai-skill, application/ai-skill+md, application/ai-skill-archive+gzip, application/agent-skills-index+json +2 more
Agent plugin10 (0.0%)1application/agent-plugins+json, application/vnd.anthropic.claude-plugin+json, application/vnd.cursor.cursor-plugin+json, application/vnd.github.copilot-plugin +4 more
Registry22 (100.0%)0none
Nested catalog33 (100.0%)0none
Types §3.3 does not cover, such as OpenAPI documents, plain JSON or Markdown, are left out rather than counted as non-conformant. Per-entry counts and every variant are in the downloadable JSON.
Notable publishers

Who has set the table.

ai-catalog.json (valid, with entries): github.com #34 · shop.app #545 · huggingface.co #1,079 · oxylabs.io #1,333 · hostinger.com #1,832 · bitdefender.com #1,844 · padlet.com #2,720 · cookieyes.com #2,749 · rudderstack.com #2,854 · zapier.com #3,005 · clerk.com #3,430 · hootsuite.com #3,449

llms.txt: cloudflare.com #2 · github.com #34 · adobe.com #61 · wordpress.org #65 · opera.com #76 · wordpress.com #99 · intuit.com #114 · samsung.com #122 · dropbox.com #136 · webex.com #146 · gravatar.com #167 · ubuntu.com #168

agents.md: shopify.com #172 · lg.com #916 · coinbase.com #1,477 · cookieyes.com #2,749 · crazyegg.com #2,988 · backblaze.com #3,334 · razorpay.com #3,369 · nazwa.pl #3,833 · nanit.com #3,942 · eufylife.com #4,848 · nownodes.io #5,585 · laravel-news.com #5,780

Your turn

Now check your own domain.

Same four files, same rules, same canary request. You get a grade against the numbers above, and the specific file you are missing. Free, no account, about ten seconds.

Edition manifest

What this edition is.

Every number on this page is anchored to one crawl, one universe and one method version. These signals measure adoption and access — a domain publishing a file, a bot being named or blocked — and they do not demonstrate citations, recommendations or visibility inside any AI system.

FieldValue
Edition2026-11
Data collected2026-11-02
Published2026-11-10
Corrected—
Methodologyv2.0
Coverage99,998 of 100,000 domains checked
UniverseTranco top-100000 (list 2026-11-01)
Own domains in scopenone
No 2026-10 edition was published. Its data was collected on 2026-10-01/02 and is archived, but it is not used as a comparison point: it recorded a one-month dip in robots.txt blocking (GPTBot fully blocked by 4,553 domains, against 8,307 in 2026-09 and 8,109 now) that this edition does not reproduce. Every change below is measured against 2026-09.
ai-catalog.json: 126 domains with a valid catalog, against 37 in 2026-09. Most of that difference is not new publishing. Of the publishers counted this month, 27 serve a file whose content first appeared after the September edition; most of the rest serve the byte-identical file we recorded in July 2026, which our crawler did not see from August until mid-October. Nothing in our crawler changed in that window, most of those domains sit behind Cloudflare, and we have not established the cause. Read the 37 of 2026-09 as an undercount, not the 126 as growth.
ard.json: 55 domains, against 4 in 2026-09. 13 serve content that first appeared after the September edition; the rest already existed in August and were not counted in September, for reasons we have not established. Same caveat as ai-catalog.json.
robots.txt: blocking of AI crawlers is flat against 2026-09 (GPTBot 8,307 to 8,109). In a sample of 94 blocking domains re-fetched on 2026-11-10, 64 carry Cloudflare's managed robots.txt block: much of the AI-crawler blocking in the top 100K is written by the CDN, not by the site. A sample, not a census figure.
New sections, measured for the first time: adoption by sector (Cloudflare content categories, multi-label), llms.txt written by the site versus served as a platform template per sector, and the media types of catalog entries against ARD v0.91 §3.3. A missing value in an earlier edition means not measured.

JSON (manifest + metrics) · CSV (by tier)

Methodology, in full: how we count.

Editions

Every month, same method.

Each edition stays at its own URL after it is replaced, so a number you cited last month is still there to check. Only editions with complete or explicitly declared coverage are published.

November 2026 (this one) · September 2026 · August 2026 · July 2026

Method

How we count.

Plain GET requests (no rendering) on each apex domain of the Tranco top-100K: /robots.txt first, then whichever of these six paths it allows: /.well-known/ai-catalog.json, /.well-known/ard.json, /.well-known/agent-card.json, /.well-known/mcp.json, /llms.txt and /agents.md. Paced at ~4 req/s globally, as DesvelaBot/0.1 (bot policy). HTML responses to text paths count as absent. Every publisher is re-probed with two canary requests to paths that cannot exist, one at the root and one under /.well-known/: servers that answer 2xx to either are excluded as catch-alls, including some seemingly legitimate publishers. Undercounting beats inflating. Separately, the MCP server cards that indexed catalogs declare are fetched to check they validate, and the DNS-AID record is looked up through a public resolver, which never touches the domain's own server.

Full disclosure, and it cuts against us on purpose: our own domains (desvela.dev, desvela.ai) publish valid manifests, and they are deliberately not in any number on this page. They are not in the Tranco top-100K, so counting them would pad the very figures we are reporting. Same rule for every other domain we index outside the list. Verify ours with a GET like any other.
Subdomains are out of scope by design (docs.example.com publishing llms.txt does not count for example.com) — the census measures apex-domain adoption, same bar for everyone.

Read the full methodology → (universe, exclusions, coverage rules and what these numbers do not claim).

Stay on the list

Get next month's census in your inbox.

One email per month with the fresh numbers and what changed: new publishers, withdrawals, bot-blocking shifts. No product spam. Unsubscribe in one click from any email. We email you a confirmation link first, and nothing is sent until you click it.