In short
- Google defines crawl budget as all the URLs it can and wants to crawl on a site.
- Its guide mainly targets sites with more than a million pages, or more than 10,000 pages that change every day, and those with many "discovered, currently not indexed" URLs.
- On an e-commerce site, the first cause of waste is faceted navigation, followed by sorting and parameters.
- The goal is not to be crawled more, but to be crawled on the right pages.
Crawl budget is a topic that worries many site owners, often unnecessarily. For most sites, Google crawls everything that deserves to be crawled without difficulty. E-commerce catalogs with faceted navigation are among the exceptions.
What crawl budget is
Google defines it as all the URLs it can and wants to crawl on a site. Since the web is practically infinite, Google limits the time and resources it spends on each site. Two factors come into play: your server's capacity to respond without slowing down, and Google's interest in your pages.
Who is concerned
Google's guide is primarily aimed at:
- large sites (more than a million unique pages) whose content changes at least every week;
- medium sites (more than 10,000 unique pages) whose content changes very quickly, every day;
- sites where a large share of URLs is classified as "Discovered, currently not indexed" in Search Console.
An e-commerce site with 3,000 products may seem far from these thresholds. But with its facets, it can generate hundreds of thousands of URLs: every combination of filters, every sort, every parameter produces a distinct address.
Sources of waste in e-commerce
| Source | Example | Treatment |
|---|---|---|
| Facet combinations with no demand | ?color=green&size=36&material=linen |
Crawling blocked, or filters as # fragments |
| Sorting | ?sort=price-ascending |
Crawling blocked |
| Technical parameters | session IDs, tracking | Crawling blocked, internal links without parameters |
| Empty pages | filter with no product | 404 status code |
| Duplicates | same product under several URLs | Consolidation (canonical, redirect) |
| Redirect chains | old URLs redirected several times | Direct redirect to the final URL |
Google reminds us in its December 2024 documentation: faceted navigation is by far the first cause of over-crawling reported by site owners.
The levers that matter
- Manage your URL inventory: block crawling of URLs that are not meant to be indexed in robots.txt.
- Consolidate duplicates, so Google does not crawl several versions of the same page.
- Keep an up-to-date sitemap, limited to indexable pages that respond with a 200 status code.
- Avoid redirect chains and server errors.
- Speed up your pages: a fast server lets Google crawl more without overloading it.
The right goal
The goal is not to be crawled more, but to be crawled on the right pages: your category pages, your facets with real demand, your product pages. Facets that deserve to be indexed must remain crawlable and well linked, as we explain in our guide to faceted navigation.
Once crawling is cleaned up, every indexed page must justify its place with tags and copy of its own: a crawled but generic page remains a weak page. That is the part unblind automates.
Frequently asked questions
Is my e-commerce site affected by crawl budget?
Probably not if you have a few thousand pages. You are affected if your catalog generates hundreds of thousands of URLs (often because of facets and sorting), or if Search Console shows a large share of "Discovered, currently not indexed" URLs.
How do I reduce crawl budget waste?
Block crawlers from valueless URLs (facet combinations with no demand, sorting, session parameters) in robots.txt, consolidate duplicates, keep an up-to-date sitemap with only indexable pages, and avoid redirect chains.