Which AI Crawlers Do the Top 1,000 Websites Block? (robots.txt Study, October 2026)
We read the robots.txt files of the 1,000 most popular websites: 20.9% block at least one of 16 AI bots. Per-bot shares, site strategies, common mistakes and open data.
On 12 October 2026 we fetched the robots.txt file of each of the 1,000 most popular websites and checked 16 AI crawlers against it. 209 of the 1,000 sites (20.9%) block at least one AI bot from the whole site; 789 (78.9%) let every one of them in. Training crawlers are blocked most often (CCBot on 15.5% of sites, GPTBot on 13.8%), bots that fetch pages for AI search less often (OAI-SearchBot on 7.0%).
Below are the five main findings, the bot-by-bot table, the strategies sites use, the mistakes we found and what it means for your own robots.txt. The per-site data and the method are open, so you can check any number.
Five key findings
- About one site in five blocks AI bots. 20.9% block at least one of the 16 AI crawlers from their home page. Only 2.0% (20 sites) block all 16.
- Training is blocked more than search. 20.6% block at least one training crawler, 12.4% at least one AI search bot and 11.8% at least one bot that fetches a page when a person asks for it.
- Half of the sites that block GPTBot still let OAI-SearchBot in: 70 of 138. For Anthropic, 58 of the 131 sites that block ClaudeBot still allow Claude-SearchBot.
- The biggest sites block most. 27% of the top 100 block at least one AI bot, against 23.8% of sites ranked 101–500 and 17.4% of sites ranked 501–1,000.
- Files drift. 109 sites still list the retired
anthropic-ainame and 94 listClaude-Web; 49 sites serve a web page instead of a robots.txt file, so none of their rules exist for a crawler.
Who we checked, in short
We walked down the Tranco research ranking (list 94GG2) and kept the first 1,000 domains that serve a real website, which took 2,364 domains: the raw list is full of infrastructure domains with no site behind them. For each site we read one robots.txt file with our own research bot, SEOpeckBot, and parsed it under RFC 9309, the robots.txt standard, using the same code as our AI Crawler Checker.
"Blocks a bot" in this study means the bot may not fetch the home page (/) while Googlebot still may. A site that only keeps a bot out of some folders is not counted as blocking. 168 sites refused our bot outright and are left out; the full method and its limits are at the end.
Three kinds of AI bot, not two
Most AI crawler studies split bots into "training" and "search". There is a third kind, and site owners treat each one differently:
- Training crawlers collect pages to build AI models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), Meta-ExternalAgent (Meta) and Amazonbot (Amazon).
Google-ExtendedandApplebot-Extendedbelong here too, although they are control tokens rather than separate crawlers: Google and Apple read them to decide whether pages fetched by their normal crawlers may be used for AI models. - AI search bots index pages so an assistant can find and cite them: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Amzn-SearchBot.
- User-triggered fetchers visit a page only when a person asks the assistant about it or pastes the link: ChatGPT-User, Claude-User, Perplexity-User and Amzn-User. Some operators say these fetchers may not apply robots.txt because a person asked for the page; Perplexity says so about Perplexity-User. Check each operator's documentation.
| Kind of bot | Bots in the study | Sites blocking at least one |
|---|---|---|
| Training | 8 | 206 (20.6%) |
| AI search | 4 | 124 (12.4%) |
| User-triggered | 4 | 118 (11.8%) |
| Any of the 16 | 16 | 209 (20.9%) |
Almost every site that blocks AI bots blocks training: only 3 of the 209 block a search or user-triggered bot without blocking any training crawler. 48 sites block all eight training crawlers.
Bot by bot: who gets blocked
The table shows, for each of the 16 AI bots, the share of the 1,000 websites that block it from the whole site, and the share whose robots.txt names it at all.
| Bot | Operator | Kind | Blocked site-wide | Named in robots.txt |
|---|---|---|---|---|
| CCBot | Common Crawl | Training | 155 (15.5%) | 17.0% |
| Bytespider | ByteDance | Training | 152 (15.2%) | 14.8% |
| GPTBot | OpenAI | Training | 138 (13.8%) | 19.5% |
| ClaudeBot | Anthropic | Training | 131 (13.1%) | 17.5% |
| Meta-ExternalAgent | Meta | Training | 118 (11.8%) | 11.6% |
| Google-Extended | Training (control token) | 112 (11.2%) | 16.2% | |
| PerplexityBot | Perplexity | AI search | 105 (10.5%) | 16.4% |
| Amazonbot | Amazon | Training | 98 (9.8%) | 10.4% |
| Applebot-Extended | Apple | Training (control token) | 97 (9.7%) | 10.4% |
| ChatGPT-User | OpenAI | User-triggered | 90 (9.0%) | 14.5% |
| Perplexity-User | Perplexity | User-triggered | 84 (8.4%) | 9.1% |
| Claude-User | Anthropic | User-triggered | 79 (7.9%) | 8.6% |
| Claude-SearchBot | Anthropic | AI search | 75 (7.5%) | 8.9% |
| OAI-SearchBot | OpenAI | AI search | 70 (7.0%) | 13.4% |
| Amzn-SearchBot | Amazon | AI search | 51 (5.1%) | 2.3% |
| Amzn-User | Amazon | User-triggered | 48 (4.8%) | 2.0% |
Two things stand out. First, naming a bot is not the same as blocking it: 195 sites name GPTBot, but 74 of them still let it fetch the home page, usually because they keep it out of some folders only. Second, Bytespider, Meta-ExternalAgent and the two Amazon AI bots are blocked by more sites than name them. That comes from "allowlist" files, explained in the next section.
For comparison, the search engines we used as controls are almost never blocked: Googlebot on 2 sites (0.2%) and Bingbot on 7 (0.7%).
Site strategies: where does your site fit?
Counting bots one by one hides how sites actually decide. So we put each site into one strategy, based on which kinds of AI bot it blocks from the home page.
| Strategy | Sites | Share |
|---|---|---|
| Open to all AI bots (no site-wide AI block) | 789 | 78.9% |
| Blocks at least one bot of each kind (training, AI search and user-triggered) | 110 | 11.0% |
| Blocks training only | 78 | 7.8% |
| Blocks training and AI search, not user-triggered | 11 | 1.1% |
| Other mix (for example search but not training) | 10 | 1.0% |
| Blocks all crawlers, Googlebot included | 2 | 0.2% |
Read the second row carefully: blocking at least one bot of each kind is not the same as blocking every AI bot. Of those 110 sites, only 20 block all 16.
About 3% of sites, 34 in all, run an allowlist: a User-agent: * group with Disallow: /, plus groups that let named search engines in. Every crawler the file does not name, AI bots included, is shut out. We count these sites as blocking, because that is what the file tells an AI bot. 33 of the 110 sites in the second row work this way, and so do 16 of the 20 sites that block all 16 bots. pinterest.com is one example: none of the 16 AI bots is named, and all of them are blocked through the * group. Other sites list the bots one by one: bbc.com blocks 14 of the 16 by name and cnn.com 15.
Block training, allow AI search: the middle path
The 78 "training only" sites are the clearest group, but the pattern runs wider. Of the 138 sites that block GPTBot, 70 (50.7%) still let OAI-SearchBot fetch the home page. Of the 131 that block ClaudeBot, 58 (44.3%) still allow Claude-SearchBot. Perplexity has no separate training crawler, and 105 sites (10.5%) block PerplexityBot.
forbes.com is one example of the approach: its file blocks GPTBot and ClaudeBot and has groups that allow OAI-SearchBot, Claude-SearchBot and Claude-User. The logic is simple: a training crawler takes content without sending visitors, while an AI search bot can cite the page and link to it. Here is a minimal version of that setup, checked with our parser:
# Block AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Amazonbot
Disallow: /
# Everyone else, including AI search and user-triggered fetchers
User-agent: *
Disallow: /admin/Bigger sites block more
The share of sites blocking AI bots falls as you go down the ranking. "Rank" here is a site's place among the 1,000 websites in the study, not its raw Tranco rank.
| Rank | Sites | Block at least one AI bot | Block GPTBot |
|---|---|---|---|
| 1–100 | 100 | 27 (27.0%) | 20 (20.0%) |
| 101–500 | 400 | 95 (23.8%) | 62 (15.5%) |
| 501–1,000 | 500 | 87 (17.4%) | 56 (11.2%) |
Drift and mistakes
robots.txt files collect old lines. These are the patterns we counted, described as neutrally as we can: some are deliberate choices we cannot see the reason for.
Legacy names
109 sites still list anthropic-ai and 94 list Claude-Web, names Anthropic no longer documents; its current bots are ClaudeBot, Claude-SearchBot and Claude-User. Other agent names outside our 16 are common too: cohere-ai appears on 115 sites, Diffbot on 106 and YouBot on 94.
Old names do little harm next to current ones, and Anthropic has said ClaudeBot honours rules written for its retired names. Only 7 sites rely on the legacy names alone, with no ClaudeBot line. We call these legacy-only rules rather than broken ones, but a parser that follows RFC 9309 to the letter would not apply them to ClaudeBot. If you want to block Anthropic's crawler, name ClaudeBot.
One operator, mixed rules
Blocking GPTBot while allowing OAI-SearchBot is a clear choice. The reverse pairing for user-triggered bots is harder to explain: 22 sites block ChatGPT-User but allow OAI-SearchBot, 7 do the same with Claude-User and Claude-SearchBot, and 7 with Perplexity-User and PerplexityBot. The search bot may index the page, but the assistant cannot open it when a person asks about it. Worth a second look if that is your file.
Lines a strict parser ignores
9 sites have lines that RFC 9309 does not recognise: 5 use a versioned name such as User-agent: GPTBot/1.0, and 4 have Allow or Disallow rules before any User-agent line, so the rules belong to no group. Some crawlers are forgiving about these, but a strict parser, ours included, skips them. Write the bare product token: User-agent: GPTBot.
A web page instead of a robots.txt file
49 sites (4.9%) answer /robots.txt with an HTML page, usually their home page or an error page sent with a 200 status. Under RFC 9309 that is not a valid robots.txt file, so a crawler finds no rules in it and treats the whole site as open. If you think you have AI rules, check that the URL returns plain text.
New signals: llms.txt and Content-Signal
- llms.txt: 192 of the 1,000 websites serve a file at
/llms.txt. We fetched it only where robots.txt allowed our bot to, which left 42 sites unchecked, so the share is 20.0% of the 958 we could check. llms.txt is a proposed format, not a standard; a file being there says nothing about whether any AI system reads it. We counted any non-empty plain-text answer and did not validate the format. - Content-Signal: 30 sites (3.0%) have a
Content-Signal:line, the format Cloudflare introduced to state preferences such asai-train=no. 15 of them sayai-train=no, but only 5 of those 15 also block GPTBot in robots.txt. A Content-Signal line states a preference; it does not block anything. - Cloudflare's policy text: 7 sites include the licence-style text of Cloudflare's Content Signals Policy in their robots.txt.
What this means for your site
- Decide per kind of bot, not per company. Blocking training says "don't learn from my pages". Blocking AI search and user-triggered fetchers also takes you out of the answers and links those assistants show. Our guide Should you block AI crawlers? walks through the trade-off.
- Use current names. Name ClaudeBot, Claude-SearchBot and Claude-User, not only
anthropic-ai. Write each name bare, without a version number. - Check what your file really does. Paste your domain into the AI Crawler Checker: it uses the same parser as this study and shows each bot as allowed, partly blocked or blocked. The Robots.txt Tester shows whether a URL is open for any user agent, and the Robots.txt Generator writes a clean file if yours has drifted.
- Remember robots.txt is a request. The large AI companies say their crawlers follow it, but it cannot stop a bot that ignores it. For a hard block, use your firewall or CDN as well.
How this compares with earlier studies
The comparison is directional only: samples and methods differ. In 2023, Originality.ai counted 242 of the top 1,000 sites blocking GPTBot within weeks of its launch; we count 138 of 1,000 websites. Cloudflare's 2025 analysis of about 3,900 robots.txt files from the top 10,000 domains found GPTBot was the most disallowed bot; in our data it ranks third, after CCBot and Bytespider. Chris Humphrey's June 2026 study of the Majestic top 10,000 found 22.6% blocking at least one of 10 bots and 4% blocking all ten; we find 20.9% blocking at least one of 16 and 2.0% blocking all 16.
Full method
Sample
The list is Tranco list 94GG2, generated on 2 October 2026 from 30 days of data (3 September to 2 October 2026). We checked domains in rank order and kept a domain as a website if its home page answered with a 2xx HTML page on the same registrable domain, stopping at 1,000 websites. That took 2,364 domains. The 1,364 skipped domains break down as: no connection 694, robots.txt or home page redirecting to another domain 221, home page not answering 2xx 189, robots.txt refused (401/403) 168, home page disallowed for our bot and not open to Googlebot 37, home page not connecting 23, robots.txt server error 12, home page not HTML 9, timeout 9, rate-limited (429) 2.
Near-duplicates are kept as Tranco lists them (for example bbc.com and bbc.co.uk, or x.com and twitter.com), so the 1,000 websites are not 1,000 separate organisations.
Fetching
The crawl ran on 12 October 2026 from our own machine, not a server. SEOpeckBot fetched /robots.txt first (up to 5 redirects; if there was no valid file, once more on the www. or bare version of the domain, which found the file for 72 sites), then the home page only if robots.txt allowed it, then /llms.txt only if allowed. 35 sites disallow our bot's home page through their * group but let Googlebot in; we counted them as websites without fetching the page. The bot sent at most 12 requests at once, waited at least 2 seconds between requests to the same host, gave up after 10 seconds without a connection or 20 seconds in total, and read at most 500 KiB of a robots.txt file. Its user agent was:
SEOpeckBot/1.0 (robots.txt study; +https://seopeck.com/page/robots-txt-study)The linked page explains the study and how to opt out. At the time of the crawl, no site had opted out or asked to be removed.
Counting
Of the 1,000 websites, 890 had a valid robots.txt, 59 had none (404 or 410), 49 served an HTML page and 2 gave another 4xx answer. Under RFC 9309 a missing or invalid file means no restrictions, so all 1,000 are in the denominator. The 168 sites that refused our bot (401 or 403) are not: a refusal says nothing about their rules. Counted as open sites instead, the headline figure would be 17.9% (209 of 1,168) rather than 20.9%.
Each file was parsed with the same code as the AI Crawler Checker, following RFC 9309: a bot obeys the group or groups that name it (case-insensitive), otherwise the * group; within a group the longest matching rule wins, and Allow wins a tie. A bot counts as blocked when / is disallowed for it. AI blocks are counted only where Googlebot may still fetch /; the 2 sites that block Googlebot as well form their own strategy. We classify Amazonbot as a training crawler, as our checker does. Legacy names such as anthropic-ai are counted when a file names them, not evaluated as live crawlers. After the crawl we re-fetched 25 random sites and re-parsed them; every result matched.
Limits
- robots.txt is a stated policy. We did not test whether bots obey it, and many sites block AI bots at the firewall or CDN without any robots.txt rule, so the real share of sites keeping AI bots out can be higher.
- It is one snapshot. robots.txt files change often; the per-site data shows each file as it was on 12 October 2026.
- We only test the home page. A site that blocks an AI bot from its articles but not from
/counts as open. - Tranco combines five popularity rankings; it is not a count of visits. The 168 sites that refused our bot are missing from the sample.
Rankings: Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczyński and Wouter Joosen. 2019. "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation." In Proceedings of the 26th Annual Network and Distributed System Security Symposium (NDSS 2019). https://doi.org/10.14722/ndss.2019.23386. List used: Tranco list 94GG2, generated on 2 October 2026, available at https://tranco-list.eu/list/94GG2.
Download the data
- Per-site CSV: one row per website with its Tranco rank, robots.txt URL and HTTP status, strategy, the result for all 16 AI bots and the two search engines, legacy names, ignored lines, Content-Signal and llms.txt.
- Summary JSON: every figure in this post.
You are welcome to use the data and charts in your own work; please credit "SEOpeck robots.txt study, October 2026" with a link to this page. We plan to repeat the study every quarter and publish what changed.
Cite this study
APA: Akram, N. (2026, October 20). Which AI crawlers do the top 1,000 websites block? (robots.txt study, October 2026). SEOpeck. https://seopeck.com/blog/ai-crawlers-robots-txt-top-1000-websites
Frequently asked questions
How many websites block GPTBot?
In our October 2026 check, 138 of the 1,000 most popular websites (13.8%) block GPTBot from their whole site. Among the top 100 it is 20 sites.
If I block GPTBot, can my pages still appear in ChatGPT search?
Blocking GPTBot does not block OAI-SearchBot, the bot OpenAI uses for search. They are separate names with separate rules, and 70 of the 138 sites that block GPTBot still allow OAI-SearchBot.
Do I still need anthropic-ai or Claude-Web in my robots.txt?
Not if you name ClaudeBot. Anthropic's current bots are ClaudeBot, Claude-SearchBot and Claude-User. The old lines do no harm, but only 7 of the 1,000 sites rely on them alone, and a strict parser would not apply them to ClaudeBot.
Does blocking AI crawlers in robots.txt actually stop them?
Only crawlers that choose to obey it. OpenAI, Anthropic, Google and Apple say their crawlers follow robots.txt, though some treat user-triggered fetchers differently. To stop a bot that ignores the file, block it at your server, firewall or CDN.
How do I check which AI bots my site blocks?
Enter your domain in the AI Crawler Checker. It reads your live robots.txt with the same parser used in this study and lists each AI bot as allowed, partly blocked or blocked.
Next step
Run your own domain through the AI Crawler Checker and compare it with the strategies above. If the result is not what you intended, rebuild the file with the Robots.txt Generator and check it again.
