UK agency use case: test before you touch production
This tester checks whether a URL is allowed or blocked by a given robots.txt content, for a specific user-agent. Paste in any robots.txt, from a live site, a staging draft, or a generated file, and enter the path you want to verify.
A common UK agency mistake: a developer cleans up a staging robots.txt (Disallow: /) before go-live but forgets to check the production file. The production copy gets overwritten, blocking Googlebot from the entire site. Running this test against the live /robots.txt content and the target URL takes seconds and prevents weeks of ranking loss.
Google's official robots.txt tester inside Search Console (Legacy Tools section) is the gold standard, but it requires account access to the property. This tool works offline, without authentication, and accepts any robots.txt content.
Disallow vs. Noindex: a key difference
| Mechanism | What it controls | Can still appear in index? |
|---|---|---|
| robots.txt Disallow | Stops the crawler visiting the URL | Yes, if linked from other sites |
| meta noindex | Crawler visits, but does not index | No, removed after next crawl |
| Both combined | Crawler cannot read the noindex tag | Yes, Disallow blocks reading noindex |
Using Disallow on a page you also want to noindex is self-defeating: Googlebot cannot read the noindex directive if it is blocked from crawling the page. Remove the Disallow and rely on noindex alone to deindex a page.
Rule matching and edge cases
Most crawlers use the longest matching path rule: the most specific matching entry wins. For Googlebot, Allow: /admin/public/ overrides a broader Disallow: /admin/. Wildcard matching (* in paths, $ for end of URL) is supported by Google and Bing but not all crawlers. Verify sitemap URLs are not accidentally blocked, a blocked sitemap slows down URL discovery significantly.
Workflow
After testing here, generate or edit the full robots.txt with the robots.txt generator, then check the HTTP response headers of your /robots.txt path using the redirect checker to confirm the file is served with a 200 status, not a redirect chain.