On this page7
The short answer
Few top brand sites block AI crawlers by name, and more than half publish a file for them. 112 of 193 have an llms.txt, and 149 of the 189 sites we could read name none of the 20 AI crawlers we looked for. 21 block one by name.
| Finding | Sites | Share |
|---|---|---|
| Publish an llms.txt | 112 of 193 | |
| Name no AI crawler in robots.txt | 149 of 189 | |
| Block at least one named AI crawler | 21 of 189 | |
| Block GPTBot in robots.txt | 9 of 189 | |
| Block ClaudeBot in robots.txt | 8 of 189 | |
| Publish llms.txt and block a named crawler | 11 of 189 | |
| Use a Content-Signal line | 31 of 193 |
How many websites publish an llms.txt?
112 of 193 (58%). Marketing and analytics tools (9 of 11) and developer tools (35 of 47) lead; media and consumer apps trail at 2 of 14. The median file is 11 KB, and 86 follow the full format: a title, a one-line summary and links.
Blocking is a different group. 6 of 10 inspiration sites and 5 of 14 media and consumer apps block a named crawler; none of the 16 AI companies or 14 fintech sites we could read do.
| Category | Sites | Publish llms.txt | Block a named crawler |
|---|---|---|---|
| Developer tools | 47 | 2 of 47 | |
| SaaS and productivity | 38 | 5 of 38 | |
| AI | 16 | 0 of 16 | |
| Design tools and site builders | 15 | 1 of 13 | |
| Fintech and payments | 15 | 0 of 14 | |
| Media and consumer apps | 14 | 5 of 14 | |
| E-commerce and brands | 12 | 1 of 11 | |
| UI and animation libraries | 12 | 0 of 12 | |
| Marketing and analytics | 11 | 1 of 11 | |
| Inspiration | 10 | 6 of 10 | |
| Security and privacy | 3 | 0 of 3 |
Which AI crawlers do websites block?
Bytespider is blocked most, by 13 of 189 sites. GPTBot follows at 9 and ClaudeBot at 8. Sites treat bots differently: 7 of the 9 that block GPTBot do not block OAI-SearchBot, the bot behind ChatGPT search.
| Crawler | Company | Mainly for | Block it | Limit paths | Named |
|---|---|---|---|---|---|
| GPTBot | OpenAI | Training | 11 | 32 | |
| OAI-SearchBot | OpenAI | AI search | 11 | 20 | |
| ChatGPT-User | OpenAI | On request | 8 | 23 | |
| ClaudeBot | Anthropic | Training | 12 | 30 | |
| Claude-SearchBot | Anthropic | AI search | 5 | 11 | |
| Claude-User | Anthropic | On request | 4 | 10 | |
| PerplexityBot | Perplexity | AI search | 12 | 27 | |
| Perplexity-User | Perplexity | On request | 4 | 9 | |
| Google-Extended | Training | 9 | 26 | ||
| Applebot-Extended | Apple | Training | 5 | 16 | |
| Meta-ExternalAgent | Meta | Training | 6 | 16 | |
| Amazonbot | Amazon | Training | 7 | 21 | |
| CCBot | Common Crawl | Training | 7 | 20 | |
| Bytespider | ByteDance | Training | 3 | 18 |
Who uses Content-Signal in robots.txt?
31 of 193 sites add a Content-Signal line, which says how they want their pages used. 22 say yes to AI training and 9 say no. 1 says no to AI answers, and 1 says no to search.
How we measured
On November 4, 2026 (UTC), we sent one GET for /robots.txt and one for /llms.txt to each brand's main address, four sites at a time, with a user agent that names Sitepeel. We followed redirects and did not retry.
The 193 sites are every brand in Sitepeel's DESIGN.md library, which leans toward tech companies. This is not the whole web, and the shares would be lower there. Sitepeel is not in the sample.
- llms.txt counts only as a real text file with a heading or links. A 404, or a redirect to an HTML page, counts as no file
- 9 sites refused or failed the llms.txt request (cash.app, codepen.io, coolors.co, descript.com, fontsinuse.com, gitlab.com, openai.com, perplexity.ai, sonos.com). They count as not publishing, so 58% is a floor
- 11 sites have no robots.txt, so they block nothing. 4 could not be read (cash.app, coolors.co, descript.com, sonos.com) and are left out
- Blocks means a group names the bot and disallows an ordinary page. A few keep small exceptions, such as the home page
- 2 sites disallow every crawler with a wildcard rule. They are not counted as blocking a named bot
- Robots.txt is a request, not a lock. We did not test firewalls or what any bot really does
- Google-Extended and Applebot-Extended are settings read from robots.txt, not separate crawlers
Download the data and cite it
The CSV has one row per brand: llms.txt result and size, robots.txt result, and the state of all 20 crawlers. Free to use with credit (CC BY 4.0). It reflects each file on the day we checked; sites change these often.
Sitepeel (2026). Who lets AI read their site? llms.txt and AI crawler rules on 193 top websites. Checked November 2026. https://sitepeel.dev/reports/llms-txt-ai-crawlers-2026/Questions
What percentage of websites have an llms.txt?
58% of the 193 top brand sites Sitepeel checked in November 2026 publish one. The sample leans toward tech companies, and we did not measure the whole web.
How many websites block GPTBot?
9 of the 189 top brand sites we could read (5%) block GPTBot in robots.txt. 11 more limit it on some paths, and 12 name it and allow it.
Which AI crawler is blocked the most?
Bytespider, blocked in robots.txt by 13 of 189 sites. Among the bots from OpenAI and Anthropic, GPTBot is at 9 and ClaudeBot at 8.
Does blocking GPTBot keep a site out of ChatGPT search?
OpenAI uses separate bots: GPTBot for training and OAI-SearchBot for ChatGPT search, each with its own rule. 7 of the 9 sites that block GPTBot do not block OAI-SearchBot.
What is an llms.txt file?
A plain text file at /llms.txt that gives AI tools a short map of a site: a title, a summary and links to the pages that matter. It is a proposed convention, not a standard.
Can I use this data?
Yes. The CSV is free to use with credit (CC BY 4.0). Link to this page as the source.
