What is crawl budget? Definition, factors and optimisation

What is crawl budget?

Crawl budget is the amount of crawling Googlebot spends on one hostname. Google treats every hostname as a separate site, so www.example.com and shop.example.com each get their own budget. When the budget is lower than the number of URLs worth crawling, new and updated pages wait longer before Google sees them, which delays indexing and rankings. Search engines such as Bing work under the same constraint, but the term is most often used in the context of Google.

How does crawl budget work?

Crawl budget works as the meeting point of what your server can handle and what Google wants to fetch. Google’s documentation describes the two parts below.

Crawl capacity limit

The crawl capacity limit is the maximum number of parallel connections and the delay between fetches that Googlebot uses on your server. Every site starts with the same conservative default. The limit goes up when the server responds quickly and goes down when it slows, returns 5xx errors or sends HTTP 429 rate-limiting signals. The limit is shared by all Google crawlers, including those for images and ads.

Crawl demand

Crawl demand is how much Google wants to crawl your URLs. It depends on 3 inputs:

• Perceived inventory: the number of URLs Google knows about, including duplicates.
• Popularity: URLs that are more popular on the web are crawled more often.
• Staleness: Google recrawls pages often enough to catch changes.

Site moves and large URL changes temporarily raise demand, because Google needs to re-crawl the new addresses.

Which websites need to manage crawl budget?

Crawl budget matters mainly for large or fast-changing websites. Google names three groups in its guide:

Google stresses that these numbers are rough estimates. A small blog whose posts are crawled on the day of publication does not need to think about crawl budget. In my e-commerce and marketplace projects, faceted navigation is the most common reason a 20,000-product shop behaves like a million-URL site.

How to optimise crawl budget

Crawl budget optimisation means removing wasted URLs and making the remaining ones cheap to fetch. Google recommends these actions:

1. Consolidate duplicate content and block unimportant URLs, such as filter combinations, in robots.txt.
2. Return 404 or 410 for permanently removed pages and fix soft 404 errors.
3. Keep XML sitemaps current, with the lastmod tag for updated pages.
4. Avoid long redirect chains, including 302 redirects that should be permanent.
5. Speed up server responses and support 304 Not Modified for unchanged resources.
6. Reduce heavy client-side rendering on pages that matter for search.

Two common myths are worth correcting. Adding noindex does not save budget, because Google still has to fetch the page to see the tag. Blocking URLs in robots.txt also does not move the budget to other pages unless Google already hits your capacity limit.

Crawl budget in SEO and AI search

Crawl budget matters for SEO because a page that is not crawled cannot be indexed, ranked or cited. Google’s AI Overviews and Google AI Mode build answers from the same index, so slow crawling also delays your presence in AI answers. International sites feel this most: every language version and every hreflang alternate is a separate URL to crawl, and a shop with 10 markets multiplies its inventory by 10. This is a standard check in my international SEO consultancy and in any e-commerce SEO audit.

To measure it, open the Crawl Stats report in Google Search Console (Settings, then Crawl stats). It shows total crawl requests, download size, average response time, responses by status code, crawl purpose (discovery or refresh) and Googlebot type, plus host status for the last 90 days.