robots.txt Tester & Checker
Fetch any site's robots.txt and test whether a URL is allowed or blocked for Googlebot and other crawlers.
What the robots.txt Tester & Checker does.
Since Google retired its own robots.txt Tester, there's no easy way to check crawl rules. Enter a site, a path, and a crawler, and Effiqs fetches the live robots.txt, parses the rules, and tells you whether that path is allowed or blocked, plus which rule decided it.
A single wrong Disallow can wipe a section out of Google. This catches it before it costs you traffic.
A site URL, a path to test, and a user-agent (Googlebot, Bingbot, GPTBot, or all).
An allowed/blocked verdict for the path, the matching rule, the declared sitemaps, and whether a robots.txt exists at all.
SEO teams making sure important pages are crawlable, and private ones aren't.
The methodology.
- We fetch /robots.txt, parse it into user-agent groups with their Allow/Disallow rules, and read any Sitemap directives.
- For the path you test, we apply Google's longest-match rule: the most specific matching pattern wins, and Allow beats Disallow on an equal-length match.
robots.txt Tester & Checker: common questions
Does robots.txt stop a page from being indexed?+
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in results (without a snippet) if it's linked elsewhere. To keep a page out of the index, allow crawling and use a noindex tag, or require authentication.