Robots.txt Tester

One wrong line in robots.txt can hide a whole section of a site from Google, and since Google retired the tester in Search Console there has been no quick way to check. Enter a site and the URLs you care about, pick a crawler, and the tester shows for each URL whether it is allowed or blocked, which user-agent group applies and the exact line that decided. Switch to "Paste rules" to try changes before you publish them. Matching follows RFC 9309 the way Google applies it, including wildcards (*) and end anchors ($).

How to use Robots.txt Tester

1. Enter the website, or switch to Paste rules to test and paste a draft robots.txt.
2. List the URLs or paths to test, one per line.
3. Choose the crawler, such as Googlebot or GPTBot, or type another user-agent token.
4. Click Test and read the result and the highlighted line for each URL.

When it comes in handy

• Checking that important pages are not blocked by accident.
• Testing a new Disallow rule before uploading it.
• Understanding why Googlebot-Image or an AI crawler follows a different group.

Frequently asked questions

Does blocking a page in robots.txt remove it from Google?

Not reliably. A blocked page can still be indexed from links, without a description. To keep a page out of search results, allow crawling and add a noindex tag. Our Noindex Checker shows all the signals.

Which rule wins when several match?

The longest matching path wins. If an Allow and a Disallow rule are the same length, Allow wins.

Why does Googlebot-Image ignore my User-agent: * rules?

A crawler uses the most specific group that names it. Googlebot-Image falls back to a Googlebot group before the * group, and ignores * completely when one exists.