Scan a site
ABCDEFGHIJKLMNOPQRSTUVWXYZ
SEO

Crawl Budget

Published: · Updated:

Crawl budget is the number of URLs a search engine bot (e.g. Googlebot) can and wants to visit on a given site within a certain time. It results from two factors: the crawl rate limit (how many requests the server can take without degrading) and crawl demand (how much the search engine "wants" to refresh and discover the site's content).

For small sites (up to a few thousand URLs) the budget is practically never a constraint. The problem starts with large stores, portals and sites with faceted navigation, where millions of URL variants can consume the budget before the bot reaches valuable pages.

What wastes crawl budget

  • Endless parameter and filter combinations (faceted navigation).
  • Duplicates without canonicalization and redirect chains.
  • Soft 404 pages and low-quality content.
  • A slow server — high TTFB lowers the crawl rate limit.

How to manage it

The basics are: an up-to-date XML sitemap with modification dates, blocking worthless paths in robots.txt, consistent canonicalization, fast server responses, and returning 404/410 for permanently removed content. Crawl stats in Google Search Console show where the budget actually goes.

Example

A store with faceted navigation generates thousands of URL variants from parameters — without blocking, the bot spends budget on filter copies instead of product pages.

URLs consuming budget
Example 
/products?color=red&size=42&sort=price
/products?color=blue&size=42&sort=price
/products?color=green&brand=nike&sort=newest
… (thousands of combinations)
Block in robots.txt
Example 
User-agent: *
Disallow: /products?*
Allow: /products/

Sources & further reading

Related terms

XML Sitemap An XML file listing a site's URLs with metadata (lastmod) that helps search engines discover and refresh pages. Defined by the sitemaps.org standard.Canonical URL The address designated as the primary version of a page among duplicates (rel="canonical"). It consolidates ranking signals and prevents copies from being indexed.Time to First Byte (TTFB) The time from sending a request until the first byte of the server response arrives. The foundation of a fast LCP; a good score is roughly ≤ 800 ms.Structured Data A standardized format for describing page content (usually JSON-LD with the Schema.org vocabulary) that helps search engines understand a page and qualify it for rich results.
Back to glossary

Check your site's technical quality

Insight analyses public SEO, performance, accessibility and security signals and turns findings into an ordered action list.

Scan a site free View a sample report
No credit card Report with date and sources No account required