Internal link audit of cometweb.io

Maciej Zmitrukiewicz · Pakiet badania

This is the material behind the article Internal link audit: find orphan pages and weak links and its Polish version, Linkowanie wewnętrzne: audyt i strony osierocone. In September 2026 I crawled my own site, cometweb.io, to count its internal links, orphan pages and click depth. No other site was crawled. A Polish summary is at the end.

What I looked at

The crawler started at the English homepage (/) and the Polish one (/pl) and followed every <a href> in the HTML the server sends. It sent one request at a time with a 0.6 s pause after each, read robots.txt first and obeyed it, and identified itself as CometWeb-LinkAudit/1.0. It kept no cookies and sent no Accept-Language, so the site's locale redirect couldn't carry over from one request to the next. It recorded any redirect hop by hop instead of following it silently, although none came up.

For every link it saved the source page, the href as written, where it resolves, the anchor text, rel, the part of the page the link sits in (header, footer, body and so on) and the target's status. Sitemap URLs the crawl never reached were requested at the end so their status is known, but their own links weren't followed, so they couldn't make themselves reachable.

A separate browser check loaded eight pages (both homepages, both blog indexes, /tools, /glossary and both versions of the article) and compared the internal link targets in the rendered page with those in the server HTML.

How I counted

• Inbound links are the number of different pages that link to a page, without self-links. A link to a redirect counts for its final URL. I counted them three ways: all links, links outside the header, top-level nav, footer and sidebar, and links in the body text only.
• An orphan page is a URL in sitemap.xml that the crawl from the two homepages never reached.
• Click depth is the shortest path in clicks from the page's own homepage: / for English and German, /pl for Polish.
• A generic anchor is an exact match against a short list in analyse.py: Google's own examples ("click here", "read more", "website", "article"), close variants such as "learn more" and "view all", and Polish equivalents such as „czytaj więcej” and „dowiedz się więcej”. "Read more about X" doesn't count.
• A parameter link points to an internal page URL with a query string.

The depth and inbound-link figures cover indexable pages only: no query string, no noindex, a canonical pointing at the page itself and outside /research/. That came to exactly the 273 sitemap URLs the crawl reached.

Results

Two things I fixed before publishing

My first crawl treated every <nav> inside the body, such as breadcrumbs, the table of contents and related posts, the same as the site header. That moved contextual links into the template bucket and made inbound links look thinner than they are. I separated in-page navigation from the header and ran the whole crawl again, and only that second run is published here.

I also compressed links.csv to links.csv.gz after the run. analyse.py reads either form and gives the same summary.json.

What this data can't tell you

• How Google sees the site. This is my crawler, not Googlebot, and I built the orphan list from the sitemap only, so a page that is neither linked nor in the sitemap stays invisible here.
• How much a link weighs. Inbound links are counts of linking pages.
• Links that appear only after you open a menu or press a button. The browser check didn't click anything.
• How the site looks later. I deploy several times a week, so running the scripts again will give different numbers.
• The three German pages are counted, but I didn't analyse them as a separate language.

Files

• summary.json: every number quoted in the article
• page-metrics.csv: inbound links, click depth and outgoing internal links for each page
• orphans.csv: the 8 orphan pages
• generic-anchors.csv: the 9 generic anchors
• redirect-links.csv: links to redirects (empty, there were none)
• render-check.json: server HTML compared with the rendered page
• links.csv.gz: every link the crawl recorded
• pages.csv, resolved-urls.csv, fetches.csv, sitemap-urls.txt, crawl-run.json: raw crawl data and the run settings
• sources.json: the outside sources I relied on, such as Google's link guidelines
• crawl.py, analyse.py, render-check.mjs: the crawler, the analysis and the browser check

To run it again, use Python 3.10 or later: python3 crawl.py, then python3 analyse.py, and optionally npm i playwright && node render-check.mjs. Crawl only sites you own or are allowed to audit.

Back to the article

Streszczenie po polsku

We wrześniu 2026 przecrawlowałem własną stronę, cometweb.io, zaczynając od strony głównej EN i PL. Brałem tylko linki <a href> z HTML-u, który zwraca serwer, wysyłałem jedno żądanie naraz z przerwą 0,6 s i trzymałem się robots.txt.

Crawl dotarł do 337 stron HTML, z czego 273 są indeksowalne, i znalazł 15 850 linków wewnętrznych. Osieroconych stron było 8: to wpisy blogowe w sitemapie, do których nie prowadzi żaden link. Linków do przekierowań i błędów było 0, wewnętrznych nofollow też 0, a żadna indeksowalna strona nie leży dalej niż 2 kliknięcia od strony głównej swojej wersji językowej. Do tego 9 ogólnych anchorów i 44 linki do adresów z parametrami.

Definicje, ograniczenia i dwie poprawki metody są opisane wyżej, a skrypty pozwalają odtworzyć każdą liczbę. Wróć do artykułu