Half the new surfaces were not there

The instrument changed this week, and the numbers got worse because of it. Every movement is now asked for a path that cannot exist before anything it publishes is believed. The raw comparison between the 8 and 12 September passes found 247 new surfaces. 79 of them sat on servers that answer as if publishing when asked for a file that is not there, and 50 more did not hold up on a second live query. 118 survived. Last week the same check passed 69% of arrivals; this week it passes 48%. The web did not get worse at publishing. We got better at not counting servers that answer anything. Among what did survive, the clearest single movement is un.org, which still serves a robots.txt and no longer names a single AI crawler in it. That is not a closure: nothing in that file blocks the whole site. It is the end of treating AI crawlers as a separate case. Counts cover only surfaces read successfully on both passes.

Started publishing 118

Domains that did not have the file on the earlier pass and did on the later one. Not "they published it this week": it is when we saw it.

Withdrawn 78

The file was there and is now gone: a 404 or 410, an empty file, or an HTML page where a text file used to be. A failed read does not count as a withdrawal, and neither does a server that answers every path we ask it for.

Edited 3614

Same file, different content. Immune to a server flapping: it only counts when both reads succeeded and the bytes differ.

Download every domain that moved (CSV) · raw JSON

How this is measured

• Two complete crawls of the Tranco top 100,000, compared domain by domain. Not a sample.
• Only surfaces read successfully on both passes count: 521,089 of them. That is the denominator for every percentage here, and it is smaller than the top 100,000 on purpose.
• 278,903 surfaces were excluded because at least one of the two reads failed. A timeout is our instrument failing, not the web changing, and counting it as a change is how this kind of number starts lying.
• 0 were not in the earlier pass at all, so they cannot be called new publishers. They are left out rather than counted.
• Servers that answer 200 to any path are flagged by a random-path probe, at the root and under /.well-known/. Every move they appear to make is dropped, in both directions: a server that answers everything is neither publishing nor withdrawing anything.

Same crawler and methodology as the monthly census. Who runs this and where the data lives: about.