Crawl budget
Crawl budget is how many URLs a search engine is willing and able to fetch from your site in a given period. Google defines it as the product of two things — a crawl capacity limit, the most it can fetch without degrading your server, and crawl demand, how much it actually wants your pages — and treats each hostname as having a budget of its own.
The two halves: capacity and demand
Capacity is a ceiling Google sets for itself. It watches how quickly your server responds and how often it errors, then backs off when the site slows down; the goal is to crawl you without knocking you over. Demand is the other half: how much Google wants your URLs at all, driven by how popular they are and how stale its copy has become. A fast server with nothing worth re-fetching still gets crawled lightly, because capacity was never the constraint.
- Raises capacity — a consistently fast, error-free response. Crawl rate follows server health, so Core Web Vitals work and cheap prerendering pay off twice.
- Lowers capacity — timeouts, 5xx errors and slow responses under load. A crawler that starts hitting errors slows down and stays slow for a while.
- Raises demand — pages that earn traffic and links, and content that genuinely changes. Sitemaps and internal links help Google find URLs, though being found is not the same as being wanted.
- Wastes both — infinite URL spaces such as faceted filters, session ids and endless calendars, plus soft 404s, redirect chains and near-duplicates that should have carried a canonical tag.
When it is a real problem, and when it is a distraction
Google's own guidance is unusually blunt about who should care. Its large-site crawl-budget guide names three audiences: sites with more than a million unique pages whose content changes about weekly, sites with more than ten thousand unique pages whose content changes daily, and sites where Search Console classifies a large share of URLs as *Discovered — currently not indexed*. If you do not fit one of those, Google's stated advice is that keeping your sitemap current and watching the Page Indexing report is adequate.
That third audience is the useful one, because it is a symptom rather than a size. It also pays to keep crawling and indexing apart: Google is explicit that not every crawled page will be indexed. A URL sitting in *Crawled — currently not indexed* was never starved of crawl budget — it was fetched, assessed and passed over, which is a content or duplication problem rather than a bandwidth one. Treating that as a crawl-budget issue is the most common way the term gets misapplied, and it sends founders tuning robots.txt when they should be consolidating thin pages.
What crawl traffic actually looks like on a long-tail site
We can put numbers on this from our own logs, and they surprised us. Our marketing site prerenders roughly three thousand pages across eight locales, squarely in the range where founders start to worry about this. Across one three-hour sample of origin requests we counted 1,438 fetches spread over 558 distinct URLs — about 2.6 fetches per URL, and for most of them that was the only visit the URL got.
The second finding mattered more. Roughly three quarters of that traffic was not a search engine at all, but datacenter scrapers with spoofed or empty user agents. Named AI crawlers we leave deliberately unblocked, because being read is the entire point of generative engine optimization. Two practical consequences follow: what looks like crawl pressure is mostly not the crawler you are optimizing for, and a short CDN cache lifetime buys a long-tail corpus very little, because each URL is requested too rarely to still be warm on the next pass. If you want to watch the search-engine half on your own site, a self-hosted monitor such as CrawlSEO reads it straight from Search Console.
FAQ
- Do I need to worry about crawl budget on a small site?
- Almost certainly not. Google points this work at sites above roughly ten thousand pages that change daily, or a million that change weekly; below that its advice is to keep the sitemap accurate, fix server errors, and spend the time on content instead. The one exception is the symptom test rather than the size test: if Search Console reports a large share of your URLs as Discovered — currently not indexed, look into it whatever your page count.
- Does blocking pages in robots.txt save crawl budget?
- It saves the fetch, but it is a blunt instrument and it does not remove a URL from the index if other pages link to it — a disallowed URL can still surface as a bare link with no description. Use it for URL spaces that are genuinely worthless, such as faceted filter combinations and internal search results. For duplicates that should consolidate, the canonical tag is the right tool; for pages you want out of the index, noindex is, and noindex only works if the crawler is still allowed to fetch the page and see it.
An AI growth team that runs this for you
AgentCeres is a managed AI marketing team — you approve what ships. 14-day free trial, from $39/month.