CometWebSEO / Website launch

Launch leftovers on 100 popular homepages

Maciej Zmitrukiewicz · Research pack for Website launch checklist: the leftovers you'll miss

This is the material behind the article. I drew domains at random from the Tranco list of the 100,000 most popular sites and checked the first 100 that served a homepage for the leftovers a launch checklist should catch: noindex, robots.txt, the sitemap, how a missing page answers, the favicon, the homepage title, the move from HTTP to HTTPS, the canonical tag and links to staging hosts.

What I looked at

The script went through the sample in order and stopped once 100 sites served a homepage. To get there it went through 167 domains. Eight workers ran in parallel, so 13 more domains past that point were already measured when it stopped; they are in the raw file with sample_index 1 and count nowhere. Ids were assigned over all 180 rows, which is why they go up to site-180. The rest were CDNs, API hosts, parked domains and pages that block bots, so every number below is out of 100. The sample is the same one I used in October, drawn with the same seed, and this time the script introduced itself as CometWebResearch/1.0 instead of pretending to be Chrome, so a few sites answered differently or not at all.

For each site the script loaded the homepage, robots.txt, the sitemap, one made-up address such as /cw-launch-check-3f9a1c/, the favicon and the plain http:// version, one request each. The robots.txt check follows Google's rules, so a group written for Googlebot wins over the * group. The staging check looks for absolute links to localhost, private endings such as .local and subdomains named staging, dev, test, preview and similar. I read every match by hand, because a subdomain called beta or demo is often a real product.

Two checks got a second pass. The first sitemap check accepted only XML, and three sites listed a plain text sitemap, which Google also accepts, so the second pass counts those as working. The favicon check ran again on every icon link for the sites that failed, and nobody passed on that try. Some of those servers answered the research user agent with 403, so the favicon number is a little high.

I split the results into two groups. Problems are things a launch checklist should have caught. The second group is worth a look but isn't an error on its own, because a site can submit its sitemap in Search Console and skip the Sitemap line, and Google picks a canonical even when the tag is missing.

The pack doesn't name the sites. Each one has a neutral id such as site-042, addresses keep only http or https, www or not and whether they stay on the same site, and homepage titles appear only when they are a default title.

Problems

What I countedNovemberOctober
Homepage with noindex (meta tag or X-Robots-Tag)11
robots.txt blocks Googlebot from the whole site11
http:// loads without moving to HTTPS86
A made-up URL answered 2001614
No favicon that loads107
Homepage title missing, empty or default33
Sitemap listed in robots.txt does not load15
Canonical points to another site, with no sign it is deliberate0new check
Canonical on http:// from an https page0new check
Sites with at least one problem31different rule

The October column uses the October data where the same check existed. The October run accepted only XML sitemaps and identified itself as Chrome, so its sitemap rows run a little high and the two columns are close but not identical tests. The canonical and staging checks are new in November.

Worth a look

What I countedNovemberOctober
No robots.txt (anything other than 200)1311
No Sitemap line in robots.txt3433
No sitemap in robots.txt, /sitemap.xml or /sitemap_index.xml2325
No canonical tag on the homepage25new check
Canonical points to another site (I read each one)3new check
Canonical on the other www variant of the same site3new check
Links to hosts that look like staging, dev or local1new check
of which I confirmed as a test environment by reading them0new check

In October the article counted seven checks, including a missing Sitemap line, and 46 sites failed at least one. The same rule on the November data gives 49. I changed the headline because a missing Sitemap line isn't an error.

What this data can't tell you

Files

To check your own list, save the domains (a client's sites, your own, a list of new launches) as top-1m.csv with one rank,domain line each, then run python3 measure.py 100, python3 recheck_sitemap.py, python3 recheck_favicon.py and python3 summarise.py.

Back to the article