robots.txt Configuration Checks and Common Mistakes
Key takeawayrobots.txt is only a few lines, yet it decides which directories crawlers may enter and whether they can read styles and scripts. One wrong Disallow can remove a whole site from the index, so this sheet covers syntax, allow scope and verification.
Industries and use cases
- Confirming crawl rules before a new site goes live
- Switching a site from staging to production
- Investigating a site that is not being indexed
- After adding admin, account or search result directories
Checklist
| Check item | Pass criteria | Risk level | Fix hint |
|---|---|---|---|
| File reachable | /robots.txt returns 200 with content type text/plain and a size under 500KB | High | A 404 or 5xx is treated as allow-all, which is riskier, so make sure the file exists. |
| Syntax and groups | Each User-agent has its own group, directives are spelled correctly and a space follows the colon | High | A misspelled directive is ignored entirely; compare against the official syntax reference line by line. |
| No site-wide block | Production does not combine User-agent: * with Disallow: / | High | Copying test rules over is the most common accident; search the whole file before launch. |
| Styles and scripts allowed | CSS, JS and image directories are not blocked | High | Blocking styles and scripts stops the renderer from judging mobile usability. |
| Admin and private paths | Admin, cart, checkout, account and search result paths are reasonably restricted | Medium | Crawling these wastes budget and can expose parameter structures. |
| Wildcard use | Wildcard and end-of-string matching behaves as intended and is not used to block a single URL | Medium | Block single URLs with noindex or a 301 instead of relying on path patterns. |
| Sitemap declaration | The sitemap is declared with an absolute URL including protocol and host | Medium | Relative paths are not recognised; multilingual sites may declare several sitemaps. |
| Environment separation | Staging and pre-production robots files differ from production, allowing crawling while serving noindex | High | The two mechanisms must work together: blocking prevents the noindex from ever being read. |
| Crawl-delay ineffective | Crawl speed is controlled by server capacity and official tools rather than Crawl-delay | Low | Major search engines ignore the directive, so writing it changes nothing. |
| Protocol and host versions | robots.txt is reachable and identical over http and https, with and without www | Medium | All four entry points should return the file rather than an error page. |
| Post-change review | Crawl stats and indexed counts are reviewed within 48 hours of a rule change | Medium | Keep a copy of the old file before major changes so you can restore quickly. |
| No internal exposure | The file contains no internal addresses, test accounts or unpublished page paths | Low | robots.txt is public; treat it as a notice, not a confidentiality tool. |
Common misjudgements
- Assuming a robots block removes a page: blocked URLs can still appear in results, and removal needs noindex or a 410.
- Blocking search result and filter parameter pages so broadly that category pages stop being indexed.
- Overlooking different files on http and https, so part of the crawling follows the wrong rules.
How to use this checklist
- Pair with the crawl budget sheet: this one covers rule writing, that one covers actual crawl behaviour.
- Re-check after adding a directory or changing routes and do a full quarterly review.
- Export crawl stats before and after the change to confirm the difference is expected.
FAQ
Can robots.txt prevent a page from being indexed?
Not completely. It blocks crawling only; removal from the index needs noindex or a 410.
Can a site have several robots.txt files?
One per host name and protocol combination; subdomains each have their own.
Do rule changes take effect immediately?
They are read at crawl time, but cached results take time to update, usually within days.
Version notes2026-11-21: first release, compiled from official Google documentation and public sources; updated as engine rules and regulations change.
If you prefer not to verify each item yourself, a pre-checked B2B template site package is available as an alternative.
Related checklists
hreflang Configuration Review for Multilingual SitesXML Sitemap Creation, Submission and ValidationCrawl Budget and Index Permission Review for New Sites404 and Redirect Configuration ReviewDuplicate Content Risk ReviewNote: this checklist is a self-check reference only. Search-engine behaviour and regulations follow the latest official documentation of each provider and jurisdiction.