Rules
UK SEO context: robots.txt mistakes Google penalises
The robots.txt file sits at the root of your domain (e.g., https://example.com/robots.txt) and tells compliant crawlers which URLs they may or may not visit. It is a convention, not a security mechanism, any bot can ignore it.
On Linux-based UK hosting (common with providers such as Fasthosts or 123-reg), file paths in robots.txt are case-sensitive. Blocking /Admin/ does not block /admin/. GOV.UK maintains a publicly documented robots.txt as a best-practice reference, allowing all crawlers while listing a clear sitemap URL.
For large UK e-commerce sites running Magento or WooCommerce, crawl budget matters: Google allocates a fixed number of requests per day per domain. Wasting budget on parameter URLs (?sort=, ?colour=) or internal search pages leaves fewer credits for product pages. Use Disallow on those paths or consolidate with canonical tags. Block Googlebot-Image only if you have genuine reasons to suppress images from UK Google Image Search.
Common directives and their effects
| Directive | Effect | SEO note |
|---|---|---|
Disallow: / | Blocks entire site | Catastrophic if applied to User-agent: * |
Disallow: /wp-admin/ | Blocks admin panel | Safe; login page still indexable |
Disallow: /*.css$ | Blocks CSS files | Breaks Google rendering; avoid |
Allow: /wp-admin/admin-ajax.php | Overrides broader Disallow | Required for WooCommerce |
Sitemap: | Declares sitemap URL | Speeds up discovery |
What robots.txt cannot do
Blocking a URL in robots.txt does not remove it from Google's index. Pages already indexed, or those linked from external sites, can still rank. To force removal, use a noindex meta tag (Googlebot must be able to crawl the page to read it) or the URL removal tool in Google Search Console. Googlebot also ignores Crawl-delay, manage crawl rate inside Search Console instead.
Workflow: generate, test, deploy
Use this generator to build your rules, then validate each URL with the robots.txt tester before uploading to your server root. After deployment, fetch the file in Google Search Console to confirm Google can read it. Check for 5xx errors on the /robots.txt path, a server error returns an empty response that some crawlers interpret as "block everything".