Free online tools

Free web tools for SEO, code and everyday tasks.

AI Crawler Checker

See which AI crawlers, such as GPTBot, ClaudeBot and Google-Extended, your robots.txt allows or blocks.


AI Crawler Checker

AI companies use their own crawlers to collect content for training models and to fetch pages for AI-powered search answers. Each one follows the rules in your robots.txt file under its own name. This tool reads your robots.txt and shows, crawler by crawler, whether it may fetch your site, so you can confirm your choices are actually in effect.

How to use AI Crawler Checker

  1. Enter a domain or a page address, such as example.com or https://example.com/blog.
  2. Click Check crawlers. The tool downloads the site's robots.txt file.
  3. Read the table: Allowed means the crawler can fetch the address you entered, Blocked means it cannot, and Partly blocked means some paths on the site are off limits to it.
  4. To change the result, edit your robots.txt. The Robots.txt Generator on this site can help you write the rules.

When it comes in handy

  • Confirming that a block you added for AI training crawlers is working.
  • Making sure you have not accidentally blocked AI search crawlers that could send you visitors.
  • Seeing how other publishers in your field handle AI crawlers.

Frequently asked questions

Does blocking a crawler in robots.txt stop it completely?
Robots.txt is a set of instructions, not a lock. The major AI companies say their crawlers respect it, but it cannot stop a crawler that chooses to ignore it. For that you need server-level blocking.
What is the difference between a training crawler and a search crawler?
Training crawlers, such as GPTBot and ClaudeBot, collect content to improve AI models. Search and user crawlers, such as OAI-SearchBot or ChatGPT-User, fetch pages to answer a question right now and can link back to you. Many sites block the first kind and allow the second.
Why are Googlebot and Bingbot in the list?
They are shown for comparison. Blocking them removes your site from Google or Bing search. Google-Extended is different: blocking it does not affect Google Search rankings.

Related Tools