What Is Crawl budget? Meaning and Example | Loop2Tech Skip to main content

New on the blog: Google Maps Ranking Factors in 2026: What Moves a Business Into the Top 3 Read it

What is Crawl budget?

Crawl budget is the number of pages a search engine will crawl on a site within a given period. Redirect chains, duplicate URLs, and slow responses waste it.

What it means in practice

Google says crawl budget is mainly a concern for very large sites, roughly a million pages or more, or sites whose pages change daily in the tens of thousands. A typical business website does not need to think about it.

Online stores are the common exception. Filters for size, colour and price can multiply 200 products into thousands of addresses, and Google may spend its visits on those instead of the pages you want ranked.

Example

Imagine a fashion store with 300 products whose collection filters combine freely. Its crawl stats would show most of Googlebot's requests going to filtered addresses, while new products wait to be indexed.

How to check it

  • In Search Console, open Settings, then Crawl stats, and see which kinds of address get crawled most.
  • Check the Pages report for large numbers of "Discovered, currently not indexed" addresses.
  • Keep filter combinations out of the index with canonical tags and robots.txt rules, and remove redirect chains.

Read our full guide on this topic

  1. Canonical tag

    A canonical tag tells search engines which URL is the primary version when similar content exists at multiple addresses, concentrating ranking signals on one page instead of splitting them.

  2. noindex

    A noindex directive tells search engines not to store a page in their index. The page must remain crawlable for the directive to be seen and obeyed.

  3. Core Web Vitals

    Core Web Vitals are Google's measured thresholds for page experience: Largest Contentful Paint for loading, Interaction to Next Paint for responsiveness, and Cumulative Layout Shift for visual stability.

  4. Largest Contentful Paint (LCP)

    Largest Contentful Paint measures how long the biggest visible element takes to render. Google treats under 2.5 seconds as good.

  5. Interaction to Next Paint (INP)

    Interaction to Next Paint measures how quickly a page responds visually after a user interacts with it. Google treats under 200 milliseconds as good.

  6. Cumulative Layout Shift (CLS)

    Cumulative Layout Shift measures how much visible content moves unexpectedly while a page loads. Google treats a score below 0.1 as good.

Every term we define is in our AEO, GEO and SEO glossary. To see how your own site does on these, run the free AI visibility checker.