The crawling capacity and demand a search engine allocates to a website over time.

Crawl budget describes the crawling resources a search engine allocates to a site over time. It involves both how much crawling the infrastructure can support and how much demand the engine has to crawl the available URLs. It is not a purchased allowance, a fixed daily quota visible to every site owner or a guarantee that fetched pages will be indexed.

Google’s detailed crawl-budget guidance is primarily for very large or rapidly changing sites and certain discovery problems. Most small sites should first check ordinary issues such as broken links, inaccessible pages and indexing directives. Calling every missing page a crawl-budget problem can distract from a more direct cause.

Where scale does matter, unnecessary URL expansion and server problems can become important. A catalogue generating huge numbers of near-empty filter combinations requires a different investigation from a ten-page business site with one blocked service page. Use actual crawl and indexing evidence to decide which situation you have.

Takeaway: Do not pay for a crawl-budget fix merely because a page has not appeared in search. Establish whether Google can discover and access it, then investigate scale-related crawling problems if the evidence supports them.

Example

A supplier’s website has several thousand products but generates far more parameter combinations through filters and repeated sorting options. The team examines those URL patterns and actual requests. A separate missing contact page is investigated directly instead of being assumed to share the same cause.

What to check

Use relevant Search Console reports and server evidence to understand discovery, requests and responses. Keep crawling separate from indexing: a successful fetch does not promise inclusion. Avoid blocking important URLs as a shortcut without understanding how the restriction affects existing content and directives.

For more detail, see Google crawl budget guidance.