Recently I ran a crawl of cometweb.io and found a mistake in this guide. Of 44 internal links from 30 pages to 37 URLs with parameters, 40 pointed to /contact?topic=… on purpose. The other 4 carried a ?source_page= tracking parameter, and two of them sat in this article's Polish version.
Every one of those URLs had a canonical to its clean version, so Google got the right signal, but my crawler still walked the needless variants. The canonical only cleaned up the symptom, and the cause was my own links.
So I start every URL with one decision: should it be its own result, a copy of another page, a redirect, or out of search?
Canonical, sitemap, robots.txt and noindex: what each one controls
A canonical URL is the address you name as the preferred version of the same or very similar content. The sitemap works on discovery and robots.txt on crawling. noindex is one of the ways to keep a public page out of results.
Google's guide on how to specify a canonical URL calls redirects and rel="canonical" strong signals, and sitemap inclusion a weak one. Google publishes no "two signals outvote one" rule, and its overview of URL canonicalization adds that content and other information count when it picks the representative of a duplicate cluster.
Canonical: consolidate URLs that show the same content
A canonical joins your URLs with the same or very similar content and leaves each one reachable. If ?utm_source=newsletter leaves the content of https://example.com/backpacks/model-a unchanged, both URLs can point to the same canonical in the <head>.
<link rel="canonical"
href="https://example.com/backpacks/model-a">Point your canonical at representative content, not at the page you'd like to promote. A product page and a broad category are both about backpacks, and they're still two different pages. RFC 6596 also allows the canonical target to sit on another domain.
Self-referencing canonical: point the preferred page at itself
A self-referencing canonical is a canonical on the preferred page that points to the page itself, and Google recommends it. A page without one can still be indexed, but then Google picks the canonical version itself, and I'd rather tell it which one I mean.
In your own configuration, use an absolute URL and one target, and avoid chains (A → B → C), loops (A → B → A) and targets that return an error.
When your CMS and SEO plugin emit different targets in the HTTP header and the HTML, it's hard to tell which you meant. Check one page's canonical and the response of its target with our canonical checker.
Canonical or redirect: redirect when the old URL should stop working
Use a permanent redirect when the old URL should stop working as a separate page. Keep canonical for variants that need to stay reachable, such as parameters that drive the interface.
According to Google's guide to redirects and Google Search, HTTP 301 and 308 move the traffic to the new address, while a canonical only names the preferred version.
Canonical for PDFs and JavaScript apps
A non-HTML document, such as a copy of a report whose preferred version lives at another address, gets its canonical in a response header.
Link: <https://example.com/report>; rel="canonical"Consolidate a PDF that sits next to your article only when it holds the same content. 5 In a JavaScript app, compare the initial HTML with the final DOM, and ideally, serve the canonical in the server response and leave it alone.
Warning: don't remove an initial noindex with JavaScript, because Google's JavaScript SEO basics say it may skip rendering once it has read that rule.
XML sitemap: list the URLs you want indexed
An XML sitemap helps most, according to Google's sitemaps overview, on large sites and where important pages are poorly linked. It works alongside internal linking, and a small, well-linked site can be discovered correctly without one.
In a live sitemap of preferred pages, I'd list only final, canonical URLs you want indexed. Each one should be reachable and indexable, with any redirect already resolved. The exception is a deliberate migration list, and one stale entry leaves the rest of your file valid.
I did this on my own site recently with 8 orphan posts, which had zero internal links pointing at them. They got noindex, follow, which keeps them out of results while crawlers still follow their links. They left the sitemap (281 → 273), and their URLs still return 200. The full story is in my internal link audit.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/backpacks/model-a</loc>
<lastmod>2026-09-10</lastmod>
</url>
<url>
<loc>https://example.com/guides/choosing-a-backpack</loc>
<lastmod>2026-09-08</lastmod>
</url>
</urlset>Sitemap limits: 50,000 URLs and 50 MB per file
A sitemap holds up to 50,000 URLs and 50 MB uncompressed, which the sitemaps.org protocol pins at 52,428,800 bytes. Gzip leaves the limit where it is, so split your longer list into files tied together with a sitemap index.
Put full URLs, protocol included, in <loc>, and keep UTF-8 and the namespace from the example. Escape an ampersand as &, so ?color=black&size=20 becomes ?color=black&size=20.
Side note. A sitemap that covers several sites needs a separate cross-site submission setup, described in Google's guide to sitemap index files, and without it, each domain gets its own sitemaps.
Sitemap lastmod should describe a real change
According to Google's guide to building a sitemap, Google uses lastmod when it's consistently and verifiably accurate for significant updates. So change it when the main content, the structured data or important links change. Bumping the copyright year doesn't count, and Google ignores <priority> and <changefreq>.
If your system doesn't know when a page changed, leave out the optional lastmod and don't stamp today's date on the whole sitemap. In a sitemap index, the element describes when the listed sitemap file changed, and the page dates live in that file. 6
Submit your sitemap in Search Console or with a Sitemap: line in robots.txt, because Google retired its old sitemap ping endpoint.
Robots.txt controls crawling for robots that follow the protocol
Robots.txt lives at /robots.txt and binds robots that follow the Robots Exclusion Protocol, as Google's introduction to robots.txt explains. It applies to one host, protocol and port.
According to Google's robots.txt specification, https://example.com/robots.txt sets no rules for https://app.example.com, and a language directory such as /de/ shares the host's file too.
This example limits crawling of a technical URL space which contains no confidential information and doesn't need noindex to be enforced. Before you use it, match the paths to your real site.
User-agent: *
Disallow: /tech-previews/
Allow: /tech-previews/instructions$
Sitemap: https://example.com/sitemap.xmlBlocking a directory is a request, and a password is a lock. A person, or a bot that ignores the protocol, can still fetch the resource, so don't protect secret URLs with robots.txt. RFC 9309 explicitly separates crawl rules from access authorisation. Check whether a specific URL on your site is blocked in our robots.txt tester.
Robots.txt rules: the most specific match wins
The most specific path match wins, whatever the order of the rules. When an Allow and a Disallow rule are equivalent, Allow wins, and paths are case-sensitive.
* matches any sequence of characters and $ marks the end of the match, so instructions$ leaves instructions?mode=print uncovered. 8 For Google, a User-agent group written for Googlebot doesn't inherit the blocks from the * group, so after adding one, review every rule for that crawler. And Google ignores crawl-delay altogether.
An empty file served with 200 allows everything, because RFC 9309 allows a file with no groups, while Disallow: / blocks the whole host.
Robots.txt outage: 404, server errors and 30 days of the old copy
For Google, a robots.txt that returns 404 means no restrictions from that file. Returning 401 or 403 for robots.txt won't limit crawling either, and 429 is the exception to how Google treats other 4xx responses.
So when robots.txt fails, note how long the outage lasted and whether Google had an earlier copy.
Noindex only works when Google can read the rule
noindex keeps a page out of results, provided the crawler fetches the resource and reads the rule. On an HTML page, put this in the <head>.
<meta name="robots" content="noindex">A PDF or any other non-HTML resource gets a response header.
X-Robots-Tag: noindexBlock the same URL in robots.txt and Google can't read the new response, so the URL may stay in results. A Noindex: line inside robots.txt itself is unsupported, as Google's guide to blocking indexing with noindex spells out.
Check both the HTML and the HTTP headers, because removing a meta tag in your CMS leaves any X-Robots-Tag from your server or CDN in place. According to Google's robots meta tag specification, a remaining noindex still applies when there's an index value somewhere else.
To remove an indexed, non-confidential page, let the crawler read noindex and confirm the result in Search Console. A crawl block you add later gives no guarantee that the page stays out.
A deleted page without an equivalent returns 404 or 410, and a page with private data goes behind access control. A temporary removal in Search Console is a stopgap. The real fix belongs on your server, as Google's guides to HTTP status codes and removing information explain.
URL parameters, pagination and translations: consolidate only true duplicates
Start with what the variant changes for your users, and I've put the question for each type in the table.
| Example | Question to ask before configuring | Sensible direction |
|---|---|---|
?utm_source=mail | Does it only tag the campaign? | For the same content, canonical to the version without tracking |
?sort=price | Does it only change the order of the same list? | Consider consolidating; check that the set of content really is comparable |
/waterproof-backpacks | Is it a useful, distinct category for a specific need? | May deserve to be indexed on its own |
?page=2 | Does it hold further products or articles? | Its own URL and canonical; do not point to page one by default |
/guide and /pl/poradnik | Is the main content translated? | Separate canonical versions plus correct language annotations |
If your filters generate a huge number of useless combinations, Google's guide to faceted navigation allows restricting crawling, and canonical is a less direct tool here. Before you copy a global block on every URL containing ?, check that it leaves your valuable pages crawlable.
Online stores produce the most of these combinations, because filters and variants stack up on paginated listings. The e-commerce SEO audit guide shows the order in which to untangle them.
For pagination, Google's pagination guide recommends a canonical of its own for each page and crawlable links onward. Redirecting the whole sequence to page one cuts your users off from later results. Translations get their own canonicals too, and according to Google's guide to localized versions, hreflang is reciprocal, includes the page itself and doesn't canonicalise the content.
Signal conflicts: diagnose against the URL's goal
Judge a signal conflict against what the URL is for. The same setup can be a bug on a landing page and correct on a technical page. The rows in the table below are hypothetical diagnostic examples.
| State | What to check | Direction of the fix |
|---|---|---|
The target page has noindex although it should be in results | Meta tag, HTTP header, environment settings | Remove the unintended block; check the response again |
A public page meant for removal has noindex and a crawl block | Can Google read the current rule? | Allow noindex to be read; for private content use access control |
The canonical points to a 404, a noindex page or another redirect | Availability and purpose of the target | Point to the right final, indexable equivalent |
| The canonical in the HTML and in the header point to different targets | Who generates each declaration? | Settle on one target and one configuration layer responsible for it |
| The live sitemap lists old redirected URLs | Is this the active sitemap or a deliberate migration list? | Update the sitemap of preferred pages; label the diagnostic list separately |
| Search Console shows a correct alternate page with a canonical | Is the canonical target the page you want indexed? | Don't undo a correct consolidation just to lower the number of excluded URLs |
| Google chose a different canonical | Content of both pages, responses, links and declarations | Judge whether the choice is wrong before you change the configuration |
According to the Page indexing report help, "Alternate page with proper canonical tag" and "Page with redirect" can be exactly what you want. Your goal is to get the important canonical pages indexed.
Besides a tag conflict, a different Google-selected canonical can come from content similarity, CMS bugs, server configuration or copies on other domains. Google's guide to fixing canonicalization issues advises first considering whether its choice makes sense for users, and I'd do that before touching any declaration.
URL migration example: old URL to new URL
In a migration, the old URL redirects permanently to the new one, and every other signal already points to the new address. In this fictional example, the content moves from /old-guide to /guides/choosing-a-backpack.
HTTP/1.1 301 Moved Permanently
Location: https://example.com/guides/choosing-a-backpack| Element | Before the fix: an example of the error | Target configuration |
|---|---|---|
| Old URL | 200 and a copy of the content with no plan to maintain it | 301 or 308 straight to the equivalent |
| New URL | 200, but a noindex left over from testing | 200, indexing allowed |
| Canonical on the new page | Points to the old URL | Points to the new URL |
| Live sitemap | Old and new URL listed as parallel targets | The new preferred URL |
| Internal links | Lead to the old path | Lead straight to the new one |
| Robots.txt | Blocks the directory with the old URLs | Lets the crawler read the redirect and its target |
Don't send many unrelated deleted pages to your home page. Google can treat such redirects as soft 404s, so match the targets or keep the correct removal status.
Google's guide to site moves with URL changes recommends keeping redirects for as long as possible, generally at least a year, and longer can help your users. It also allows a separate old sitemap for tracking the old set of URLs. Redirect warnings in it are expected, as long as you keep it apart from the live sitemap.
How to check canonical, sitemap and robots.txt in 6 steps
Here's the order I'd go in, because each step narrows the list of causes.
Step 1: Write down the expected state of each URL
Before you touch any configuration, write down what the URL is for: indexing, consolidation, redirect, exclusion or removal. Skip that and it's easy to "fix" a correct redirect by giving the old URL a 200 status again.
Next to the goal, note the preferred URL, along with the date and release version.
Step 2: Read the HTTP response and the HTML
The command saves the headers and body of the first response, and deliberately skips following redirects, so replace the fictional URL with yours.
curl --max-time 20 --silent --show-error \
--dump-header headers.txt \
--output page.html \
'https://example.com/guides/choosing-a-backpack'Check the status, Location, Link and X-Robots-Tag, then look at the meta robots tag and the canonical in the HTML, and in the rendered DOM for a JavaScript app. Repeat the check for the redirect target, because recording only the final 200 hides problems in the earlier responses.
Step 3: Compare URL Inspection index data with the live test
In the URL Inspection tool, check the last crawl and whether indexing is allowed. Then compare the canonical you declared with the one Google selected.
According to Search Console's URL Inspection help, the Google-selected canonical comes from index data, and the live URL test doesn't predict that choice. So after a fix, the live test can look fine while the index still describes an older state.
Read "URL is available to Google" as availability only. Record the dates of both readings, and mark missing data as unknown.
Step 4: Read the Sitemaps and robots.txt reports
The Sitemaps report shows submission history and problems fetching and processing sitemaps. Whether the listed pages are indexed is a separate check.
The robots.txt report in Search Console settings shows the files Google found, with fetch dates and errors. In urgent cases it lets you request a recrawl. The old rule editor is gone, so compare the rule for a specific URL with the URL Inspection result.
Step 5: Separate a technical error from an indexing decision
"Crawled - currently not indexed" means Google fetched the page, so your robots.txt let it through. 9 Before you flip the canonical, look at the page's content, its role and similar URLs.
On a large site, add your server logs and crawl reports. According to Google, crawl budget matters mainly for very large or fast-changing sets of URLs, so on a small site I'd look for the cause elsewhere.
Step 6: Record the fix and monitor the same set of URLs
For your important URLs you can request reindexing, and for a larger set, rely on your sitemap. Google's page on how to ask Google to recrawl your URLs says requests have a quota and give no guarantee of indexing. Resubmitting the same URL won't speed anything up.
If nothing has changed after an agreed four or six weeks, that justifies a fresh diagnosis, with a signal conflict as one hypothesis among several. Compare the same URLs against the same expected states. In a migration, keep old and new URLs apart, because otherwise a correct rise in redirects looks like your site getting worse.
Indexing signals checker: line up one URL before handing over a fix
The helper checks whether your declarations agree, using only data you enter by hand. Enter the target URL that should be indexed, meaning the new address after the redirect.
Indexing signals checker
Line up the HTTP status, canonical, sitemap inclusion, noindex, robots.txt and internal linking for one URL. Confirm Google's own choice separately in Search Console.
You'll cover a larger set of URLs faster with a crawl. The SEO module in CometWeb Insight, our website audit app, checks canonical and HTTP status, plus crawl access and indexing directives. The Crawl module finds public URLs through links and the sitemap or from your list, then checks their HTTP codes and redirect chains.
The methodology has the details, and you should read the report next to your Search Console data. If you'd like to run this checklist across your whole site, create an Insight account and start your first audit.
Three takeaways on indexing signals
- The URL's goal comes before the syntax. The syntax is the easy part, and the hard part is deciding what the URL should be before you change anything.
- Clean reports remove specific obstacles. After that, Google processes the content separately and picks the pages it indexes itself.
- Check one important URL tomorrow morning. Write down its goal and run it through the six steps, then come back to it in Search Console a few weeks later.
All four signals should say the same thing about it.
Frequently asked questions
Does every page need a canonical?
It doesn't have to. Google can choose a canonical version without an explicit declaration. A self-referencing canonical is a recommended practice that makes your own preference easier to maintain, and indexing is still Google's call.
Does robots.txt keep a page out of Google?
Robots.txt only controls whether a crawler may fetch a URL, so a blocked URL can still appear in results if Google learns about it from links. When the goal is to keep a page out of search, use noindex on a crawlable page, restrict access, or remove the resource.
Does canonical work across domains?
A cross-domain canonical works, and there is no basis for a universal rule that it is weaker just because the target is on another domain. It still has to reflect the relationship between the content. For syndication, Google does not recommend relying on canonical to prevent duplication by partners. It names blocking indexing of the copy as the more effective route.
Does a sitemap have to be indexed as a search result?
A sitemap only has to be fetchable and processed correctly, so it never needs to show up as a search result. Judge that in the Sitemaps report and check indexing of the pages it lists separately. A sitemap missing from search results is a different thing from a fetch error.
What does a successfully processed sitemap actually get you?
A successfully processed sitemap means the URLs it lists can be queued for crawling. Submitting it helps Google discover those URLs, but it does not guarantee indexing. Whether each page is crawled and indexed is decided separately.
Sources for these answers: canonical URL, robots.txt, blocking indexing, RFC 6596, fixing canonicalization, sitemaps and the Sitemaps report.
Sources and scope
The article is mainly about Google Search. The RFCs and the sitemap protocol set out broader rules, and implementation details can differ between crawlers. The example.com examples are fictional.
- Google Search Central: How to specify a canonical URL.
- Google Search Central: Introduction to robots.txt.
- Google Search Central: Block Search indexing with noindex.
- Google Search Central: Learn about sitemaps.
- RFC Editor / IETF: RFC 6596: The Canonical Link Relation.
- Sitemaps.org: Sitemaps XML format.
- Google Crawling Infrastructure: How Google interprets the robots.txt specification.
- RFC Editor / IETF: RFC 9309: Robots Exclusion Protocol.
- Google Search Console Help: Page indexing report.
- Google Search Central: Fix canonicalization issues.
- Google Search Console Help: Sitemaps report.
- HTTP Archive: Web Almanac 2025: SEO.