Crawl Budget and Index Permission Review for New Sites
Key takeawayA new site has a limited crawl allowance. If parameter and duplicate pages consume most of it, the product pages that need indexing wait in line. This sheet reviews server response, rule conflicts and URL structure to keep crawling focused.
Industries and use cases
- Observing crawl behaviour in the first month after launch
- When crawl volume is high but indexed pages stay low
- After the number of pages grows quickly
- When crawling breaks after adding a CDN or firewall
Checklist
| Check item | Pass criteria | Risk level | Fix hint |
|---|---|---|---|
| Server response time | Average response under 500ms and 95th percentile under 1.5s | High | Slow responses directly cut crawl frequency; fix database queries and image weight first. |
| Error rate | 5xx responses below 1% and 429 responses are not sustained | High | Frequent timeouts trigger throttling; check whether rate limits are hitting legitimate crawls. |
| Rule conflicts | No URL is both blocked in robots and submitted in the sitemap | High | Contradictory rules make the system abandon the directory, so generate both from one configuration. |
| Index permissions | Directories that must be indexed are not noindexed and templates do not output noindex site-wide | High | This is easily added during a redesign; check page source after release. |
| Parameter control | Filter, sort and pagination parameters do not create endless combinations and important ones have canonical URLs | High | Canonicalise or restrict combinations that have no standalone value. |
| Click depth | Important pages are reachable within 3 clicks from the home page | Medium | Deep pages are crawled less often; shorten paths with category and related product links. |
| Internal link coverage | Every listed product is linked from at least one listing page | Medium | Scan for orphan pages and add unlinked products to categories or topic pages. |
| Rendering dependency | Core content does not depend on client-side async loading | High | Server-rendered content avoids the second render queue and crawls more efficiently. |
| Firewall and CDN | Search engine IP ranges are not blocked and static assets are not behind forced verification | High | After enabling bot protection, allow known crawlers or expect widespread 403 responses. |
| Sitemap freshness | The sitemap updates within 24 hours of a content change | Medium | Connect generation to the publishing workflow to reduce manual misses. |
| Crawl monitoring | Weekly records of crawl requests, average response time and status code distribution | Medium | A baseline is the only way to judge whether a change caused an anomaly. |
| Low-value pages | Tag pages, empty categories and search result pages are not widely crawlable | Medium | These consume budget and rarely add value, so restrict them as needed. |
Common misjudgements
- Treating a drop in crawl volume as a penalty: most cases trace to slower pages or a rule change, so check response time and status codes first.
- Watching only the total indexed count: a hundred indexed category pages with no product pages means the budget went to the wrong place.
- Submitting a huge URL list right after launch, when a new site with low trust is better served by batches.
How to use this checklist
- Combine with the robots and sitemap sheets as a crawl trio: this one reads results, those two cover configuration.
- Review weekly during the first month, then monthly once stable.
- Pair with the page and content sheets so quality is in place before competing for crawl allowance.
FAQ
Is low crawl frequency normal for a new site?
Yes. Trust builds over time, and steady updates plus fast responses gradually increase it.
What if the site has been throttled?
First confirm whether your own responses time out or return errors; after fixing them, recovery usually takes a few weeks.
Should every new page be submitted manually?
No. A sitemap update plus internal links is enough; manual submission is for priority pages only.
Version notes2026-11-21: first release, compiled from official Google documentation and public sources; updated as engine rules and regulations change.
If you prefer not to verify each item yourself, a pre-checked B2B template site package is available as an alternative.
Related checklists
hreflang Configuration Review for Multilingual Sitesrobots.txt Configuration Checks and Common MistakesXML Sitemap Creation, Submission and Validation404 and Redirect Configuration ReviewDuplicate Content Risk ReviewNote: this checklist is a self-check reference only. Search-engine behaviour and regulations follow the latest official documentation of each provider and jurisdiction.