Launch leftovers on 100 popular homepages
This is the material behind the article. I drew domains at random from the Tranco list of the 100,000 most popular sites and checked the first 100 that served a homepage for the leftovers a launch checklist should catch: noindex, robots.txt, the sitemap, how a missing page answers, the favicon, the homepage title, the move from HTTP to HTTPS, the canonical tag and links to staging hosts.
What I looked at
The script went through the sample in order and stopped once 100 sites served a homepage. To get there it went through 167 domains. Eight workers ran in parallel, so 13 more domains past that point were already measured when it stopped; they are in the raw file with sample_index 1 and count nowhere. Ids were assigned over all 180 rows, which is why they go up to site-180. The rest were CDNs, API hosts, parked domains and pages that block bots, so every number below is out of 100. The sample is the same one I used in October, drawn with the same seed, and this time the script introduced itself as CometWebResearch/1.0 instead of pretending to be Chrome, so a few sites answered differently or not at all.
For each site the script loaded the homepage, robots.txt, the sitemap, one made-up address such as /cw-launch-check-3f9a1c/, the favicon and the plain http:// version, one request each. The robots.txt check follows Google's rules, so a group written for Googlebot wins over the * group. The staging check looks for absolute links to localhost, private endings such as .local and subdomains named staging, dev, test, preview and similar. I read every match by hand, because a subdomain called beta or demo is often a real product.
Two checks got a second pass. The first sitemap check accepted only XML, and three sites listed a plain text sitemap, which Google also accepts, so the second pass counts those as working. The favicon check ran again on every icon link for the sites that failed, and nobody passed on that try. Some of those servers answered the research user agent with 403, so the favicon number is a little high.
I split the results into two groups. Problems are things a launch checklist should have caught. The second group is worth a look but isn't an error on its own, because a site can submit its sitemap in Search Console and skip the Sitemap line, and Google picks a canonical even when the tag is missing.
The pack doesn't name the sites. Each one has a neutral id such as site-042, addresses keep only http or https, www or not and whether they stay on the same site, and homepage titles appear only when they are a default title.
Problems
| What I counted | November | October |
|---|---|---|
| Homepage with noindex (meta tag or X-Robots-Tag) | 1 | 1 |
| robots.txt blocks Googlebot from the whole site | 1 | 1 |
| http:// loads without moving to HTTPS | 8 | 6 |
| A made-up URL answered 200 | 16 | 14 |
| No favicon that loads | 10 | 7 |
| Homepage title missing, empty or default | 3 | 3 |
| Sitemap listed in robots.txt does not load | 1 | 5 |
| Canonical points to another site, with no sign it is deliberate | 0 | new check |
| Canonical on http:// from an https page | 0 | new check |
| Sites with at least one problem | 31 | different rule |
The October column uses the October data where the same check existed. The October run accepted only XML sitemaps and identified itself as Chrome, so its sitemap rows run a little high and the two columns are close but not identical tests. The canonical and staging checks are new in November.
Worth a look
| What I counted | November | October |
|---|---|---|
| No robots.txt (anything other than 200) | 13 | 11 |
| No Sitemap line in robots.txt | 34 | 33 |
| No sitemap in robots.txt, /sitemap.xml or /sitemap_index.xml | 23 | 25 |
| No canonical tag on the homepage | 25 | new check |
| Canonical points to another site (I read each one) | 3 | new check |
| Canonical on the other www variant of the same site | 3 | new check |
| Links to hosts that look like staging, dev or local | 1 | new check |
| of which I confirmed as a test environment by reading them | 0 | new check |
In October the article counted seven checks, including a missing Sitemap line, and 46 sites failed at least one. The same rule on the November data gives 49. I changed the headline because a missing Sitemap line isn't an error.
What this data can't tell you
- These are popular sites that have been live for a while, so they show what survives a launch, and a fresh launch is a different moment.
- Tranco's top 100,000 includes expired domains now used for spam. I kept them, because skipping them by hand would make the sample less random.
- I checked homepages only. Other templates on the same site can fail checks the homepage passes.
- The title, noindex and canonical checks read the HTML the server sends, so anything added by JavaScript isn't counted.
- I measured once, from one connection in Poland. A site that was down for a moment counts as failing, and an honest user agent gets blocked more often than a browser one.
Files
checklist-en.md: the handover record from the article, in English, ready to copychecklist-pl.md: the same record in Polishresults.csv: one row per site with every check, by id and without addressesresults.jsonl.gz: raw results for all 180 domains the script measured, including the ones that served no page and the 13 past the cut (sample_index1)review.json: my notes on every staging-looking link, by idsummary.json: the counts used in the articlemeasure.py: the measuring scriptrecheck_sitemap.py: second sitemap pass that also accepts plain text sitemapsrecheck_favicon.py: second favicon pass that tries every icon linksummarise.py: turns results.jsonl into summary.json and results.csvoctober/: the earlier run on the same sample, with its own scripts and data
To check your own list, save the domains (a client's sites, your own, a list of new launches) as top-1m.csv with one rank,domain line each, then run python3 measure.py 100, python3 recheck_sitemap.py, python3 recheck_favicon.py and python3 summarise.py.