By Nadeem Akram · Sep 23, 2026

Which AI crawlers visit websites, what the difference between training and search bots is, and how to allow or block each one with robots.txt.

Alongside Googlebot and Bingbot, most websites are now visited by crawlers from AI companies. Some collect text to train language models; others fetch pages so an AI assistant can answer a question and cite its source. Site owners are increasingly asking the same question: should I let them in?

There is no single right answer. It depends on what you publish and what you want in return. This guide explains the main crawlers, what they are used for, and how to control them.

Two kinds of AI crawler

Training crawlers collect content to build and improve AI models. Examples are OpenAI's GPTBot, Anthropic's ClaudeBot, Meta's Meta-ExternalAgent and Common Crawl's CCBot, whose open dataset is used by many AI projects. Blocking these keeps your future content out of new training runs. It does not remove anything that was collected before.

Search and user crawlers fetch pages in response to what people ask right now. Examples are OAI-SearchBot and ChatGPT-User from OpenAI, Claude-SearchBot and Claude-User from Anthropic, and PerplexityBot. These are the bots that can show your page as a source and send visitors to it.

There are also control tokens that are not separate crawlers at all. Google-Extended lets you opt out of your content being used for Google's Gemini models without affecting Google Search. Applebot-Extended does the same for Apple's AI training.

How robots.txt rules work

robots.txt is a plain text file at the root of your site, for example https://example.com/robots.txt. It is made of groups. Each group names one or more crawlers with User-agent lines, followed by Allow and Disallow rules:

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

Two details trip people up:

• A crawler follows only the most specific group that names it. In the example above, GPTBot obeys its own group and ignores the * group completely. A crawler that is not named anywhere falls back to *.
• Within a group, the longest matching rule wins. Disallow: / with Allow: /blog/ means the blog is allowed and everything else is blocked.

Common setups

Block AI training, allow AI search. This is a popular middle ground for publishers who want citations and visits but do not want their work used to train models:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

Allow everything. If you want the widest possible reach, including being part of the data AI tools learn from, you do not need any AI-specific rules at all.

Block everything AI. Add the search and user crawlers to the list above as well. Be aware that you will no longer appear as a source in those assistants' answers.

What robots.txt cannot do

robots.txt is a request, not a wall. The large AI companies say their crawlers respect it, but a badly behaved bot can ignore it. If you need to stop a crawler for certain, block it at the server or firewall level as well. Also remember that robots.txt is public: anyone can read it, so never use it to hide private areas. Protect those with a login.

Check what your site allows today

It is easy to get a rule slightly wrong, for example by adding a crawler to a group you did not intend, or leaving a typo in its name. The AI Crawler Checker reads your live robots.txt and shows, crawler by crawler, whether it is allowed, blocked or partly blocked. If you need to write or rebuild the file, the Robots.txt Generator gives you a starting point, and the On-Page SEO Checker confirms your important pages are still open to search engines afterwards.