How to test a robots.txt file
- Fetch a site's robots.txt, or paste the file.
- Enter the page address or path to test, and choose a crawler.
- Read the result and the line that decides it.
Examples
A shop blocks its cart. With User-agent: * and Disallow: /cart/, testing https://shop.example/cart/checkout for Googlebot gives Blocked by line 2. Testing /cartoons gives Allowed, because the rule ends with a slash and /cartoons doesn't start with /cart/.
Allow wins a tie. With Allow: /folder and Disallow: /folder in the same group, /folder/page is Allowed. Both paths are 7 characters long, so Google uses the less restrictive rule.
A wildcard with an end mark. Disallow: /*.pdf$ blocks /guides/price-list.pdf but not /guides/price-list.pdf?download=1, because $ means the address must end right there.
Separate groups. A file with a group for GPTBot and ClaudeBot that says Disallow: /, and a * group that blocks only /search, lets Googlebot crawl /blog/ while both AI crawlers are blocked from every page.
About this tool
Check what a robots.txt file really allows before you rely on it. Paste the file or fetch it from a site, enter a page address and pick a crawler. The tool applies Google's documented rules: the group with the most specific user agent is used, the longest matching rule wins, Allow wins a tie, and * and $ work as wildcards. It marks the deciding line and flags lines that crawlers ignore.
Common questions
Why does Allow win when two rules match?
Google uses the most specific rule, which is the one with the longest path. When an Allow rule and a Disallow rule are the same length, Google picks the less restrictive one, which is Allow.
Does a blocked page stay out of Google?
Not always. robots.txt stops crawling, not indexing. Google can still list a blocked address, without a description, if other pages link to it. To keep a page out, let it be crawled and add a noindex tag.
Why is GPTBot blocked when Googlebot isn't?
Each crawler follows only the group that names it, or the * group if none does. A group for GPTBot doesn't affect Googlebot, and a crawler with its own group ignores the * rules.
Why does Google ignore my crawl-delay line?
Google reads only user-agent, allow, disallow and sitemap lines. Bing and some other crawlers do read crawl-delay.
What if robots.txt returns an error?
A 4xx error other than 429 means Google crawls as if there were no file. A 5xx or 429 error makes Google stop crawling the site for a while. The tool tells you which case applies.
Is it free to use?
Yes. There's no sign-up for everyday use, and a free account raises the daily limit if you need more.
Do I need to install anything?
No. It works in your browser on a computer or a phone.
Is what I enter kept?
It's only used to get your result. Some public answers, such as the branches for a pincode, are kept for a while so the next lookup is quicker, but nothing links them to you.