LEYUN · SITE LAUNCH CHECKLIST独立站上线自查清单库

robots.txt Configuration Checks and Common Mistakes

Sections: Crawl & Indexing · Reviewed: 2026-11-21 · 12 items

Key takeawayrobots.txt is only a few lines, yet it decides which directories crawlers may enter and whether they can read styles and scripts. One wrong Disallow can remove a whole site from the index, so this sheet covers syntax, allow scope and verification.

Industries and use cases

  • Confirming crawl rules before a new site goes live
  • Switching a site from staging to production
  • Investigating a site that is not being indexed
  • After adding admin, account or search result directories

Checklist

Checklist: robots.txt configuration and common errors
Check itemPass criteriaRisk levelFix hint
File reachable/robots.txt returns 200 with content type text/plain and a size under 500KBHighA 404 or 5xx is treated as allow-all, which is riskier, so make sure the file exists.
Syntax and groupsEach User-agent has its own group, directives are spelled correctly and a space follows the colonHighA misspelled directive is ignored entirely; compare against the official syntax reference line by line.
No site-wide blockProduction does not combine User-agent: * with Disallow: /HighCopying test rules over is the most common accident; search the whole file before launch.
Styles and scripts allowedCSS, JS and image directories are not blockedHighBlocking styles and scripts stops the renderer from judging mobile usability.
Admin and private pathsAdmin, cart, checkout, account and search result paths are reasonably restrictedMediumCrawling these wastes budget and can expose parameter structures.
Wildcard useWildcard and end-of-string matching behaves as intended and is not used to block a single URLMediumBlock single URLs with noindex or a 301 instead of relying on path patterns.
Sitemap declarationThe sitemap is declared with an absolute URL including protocol and hostMediumRelative paths are not recognised; multilingual sites may declare several sitemaps.
Environment separationStaging and pre-production robots files differ from production, allowing crawling while serving noindexHighThe two mechanisms must work together: blocking prevents the noindex from ever being read.
Crawl-delay ineffectiveCrawl speed is controlled by server capacity and official tools rather than Crawl-delayLowMajor search engines ignore the directive, so writing it changes nothing.
Protocol and host versionsrobots.txt is reachable and identical over http and https, with and without wwwMediumAll four entry points should return the file rather than an error page.
Post-change reviewCrawl stats and indexed counts are reviewed within 48 hours of a rule changeMediumKeep a copy of the old file before major changes so you can restore quickly.
No internal exposureThe file contains no internal addresses, test accounts or unpublished page pathsLowrobots.txt is public; treat it as a notice, not a confidentiality tool.

Common misjudgements

  • Assuming a robots block removes a page: blocked URLs can still appear in results, and removal needs noindex or a 410.
  • Blocking search result and filter parameter pages so broadly that category pages stop being indexed.
  • Overlooking different files on http and https, so part of the crawling follows the wrong rules.

How to use this checklist

  • Pair with the crawl budget sheet: this one covers rule writing, that one covers actual crawl behaviour.
  • Re-check after adding a directory or changing routes and do a full quarterly review.
  • Export crawl stats before and after the change to confirm the difference is expected.

FAQ

Can robots.txt prevent a page from being indexed?
Not completely. It blocks crawling only; removal from the index needs noindex or a 410.
Can a site have several robots.txt files?
One per host name and protocol combination; subdomains each have their own.
Do rule changes take effect immediately?
They are read at crawl time, but cached results take time to update, usually within days.
Version notes2026-11-21: first release, compiled from official Google documentation and public sources; updated as engine rules and regulations change.

If you prefer not to verify each item yourself, a pre-checked B2B template site package is available as an alternative.

Note: this checklist is a self-check reference only. Search-engine behaviour and regulations follow the latest official documentation of each provider and jurisdiction.