Robots.txt Tester
Robots.txt Tester
One wrong line in robots.txt can hide a whole section of a site from Google, and since Google retired the tester in Search Console there has been no quick way to check. Enter a site and the URLs you care about, pick a crawler, and the tester shows for each URL whether it is allowed or blocked, which user-agent group applies and the exact line that decided. Switch to "Paste rules" to try changes before you publish them. Matching follows RFC 9309 the way Google applies it, including wildcards (*) and end anchors ($).
How to use Robots.txt Tester
- Enter the website, or switch to Paste rules to test and paste a draft robots.txt.
- List the URLs or paths to test, one per line.
- Choose the crawler, such as Googlebot or GPTBot, or type another user-agent token.
- Click Test and read the result and the highlighted line for each URL.
When it comes in handy
- Checking that important pages are not blocked by accident.
- Testing a new Disallow rule before uploading it.
- Understanding why Googlebot-Image or an AI crawler follows a different group.
Frequently asked questions
Does blocking a page in robots.txt remove it from Google?
Which rule wins when several match?
Why does Googlebot-Image ignore my User-agent: * rules?
Related guides
-
Which AI Crawlers Do the Top 1,000 Websites Block? (robots.txt Study, October 2026)
We read the robots.txt files of the 1,000 most popular websites: 20.9% block at least one of 16 AI bots. Per-bot shares, site strategies, common mistakes and open data.
-
What CMS Is This Website Using? How to Tell in Seconds
Find out what CMS any website is built with: paste the address into a CMS detector, or check the page source, headers and robots.txt yourself.
-
How to Check Any Page's SEO in 10 Seconds (and What Each Result Means)
Check a page's title, description, H1, canonical, noindex, robots.txt, alt text, schema, Open Graph and speed in seconds, and know which results matter.