noindex vs other signals
- noindex, follow: indexing exclusion without an additional nofollow hint; it does not promise ongoing crawling or link-value transfer.
- noindex, nofollow: indexing exclusion plus a hint about following links; it is not access control.
- robots.txt Disallow: blocks crawling; does not guarantee removal of URLs already indexed.
- Do not combine noindex with a canonical pointing elsewhere. Those are conflicting signals for the bot.
Best practices
- Record why each page group should be excluded; do not apply noindex to all filters or parameters.
- Remove noindexed URLs from the XML sitemap, because consistent signals make crawling easier.
- Protect private staging with authentication. Noindex may be an additional signal on public previews, but does not protect their content.
- After removing noindex in production, check GSC for re-indexing. It does not happen instantly.
Common mistakes
Accidental noindex on a template (e.g. after a staging deploy with forgotten meta) is one of the most common SEO regressions. Technical monitoring and meta robots audits catch it before organic traffic drops. noindex alone does not instantly remove a page from the index. The bot must revisit and read the directive.
Example
A generic thank-you page is public and should be excluded from results. The bot must be able to fetch the response containing the directive. Do not expose private submission data on that page.
<meta name="robots" content="noindex, follow">X-Robots-Tag: noindex, nofollowThe directive must be fetched and processed. There is no guaranteed deadline. Check crawler access and the current state in URL Inspection; a recrawl request does not guarantee faster processing.
No. Robots.txt controls crawling and can prevent reading noindex. Authentication protects access. HTTP statuses describe different conditions: an arbitrary non-200 status is not a universal crawl block.
Technically yes, but it should not be. A sitemap suggests URLs to index; noindex says the opposite. Keep those signals aligned.