# llms.txt and AI crawlers on 193 top sites > Sitepeel checked 193 top brand sites in November 2026: 58% publish an llms.txt and 11% block a named AI crawler in robots.txt. Bot-by-bot table, free CSV. Canonical page: https://sitepeel.dev/reports/llms-txt-ai-crawlers-2026/ Sitepeel measured 193 top brand websites in November 2026: 58% publish an llms.txt, and 11% block at least one named AI crawler in robots.txt. GPTBot is blocked in robots.txt by 5% of sites and ClaudeBot by 4%. Most sites that block a bot still leave AI search bots unblocked. Published: 2026-11-05 Facts and prices checked: 2026-11-04 ## In short - 112 of 193 sites publish an llms.txt, led by marketing and developer tools - 21 of 189 block at least one named AI crawler in robots.txt - 9 block GPTBot and 8 block ClaudeBot; Bytespider is the most blocked, at 13 - 17 of the 21 block training bots but no AI search bot ## The short answer Few top brand sites block AI crawlers by name, and more than half publish a file for them. 112 of 193 have an llms.txt, and 149 of the 189 sites we could read name none of the 20 AI crawlers we looked for. 21 block one by name. 193 brand websites, checked November 2026 | Finding | Sites | Share | | --- | --- | --- | | Publish an llms.txt | 112 of 193 | 58% | | Name no AI crawler in robots.txt | 149 of 189 | 79% | | Block at least one named AI crawler | 21 of 189 | 11% | | Block GPTBot in robots.txt | 9 of 189 | 5% | | Block ClaudeBot in robots.txt | 8 of 189 | 4% | | Publish llms.txt and block a named crawler | 11 of 189 | 6% | | Use a Content-Signal line | 31 of 193 | 16% | ## How many websites publish an llms.txt? 112 of 193 (58%). Marketing and analytics tools (9 of 11) and developer tools (35 of 47) lead; media and consumer apps trail at 2 of 14. The median file is 11 KB, and 86 follow the full format: a title, a one-line summary and links. Blocking is a different group. 6 of 10 inspiration sites and 5 of 14 media and consumer apps block a named crawler; none of the 16 AI companies or 14 fintech sites we could read do. By category, checked November 2026 | Category | Sites | Publish llms.txt | Block a named crawler | | --- | --- | --- | --- | | Developer tools | 47 | 35 (74%) | 2 of 47 | | SaaS and productivity | 38 | 22 (58%) | 5 of 38 | | AI | 16 | 9 (56%) | 0 of 16 | | Design tools and site builders | 15 | 9 (60%) | 1 of 13 | | Fintech and payments | 15 | 6 (40%) | 0 of 14 | | Media and consumer apps | 14 | 2 (14%) | 5 of 14 | | E-commerce and brands | 12 | 5 (42%) | 1 of 11 | | UI and animation libraries | 12 | 8 (67%) | 0 of 12 | | Marketing and analytics | 11 | 9 (82%) | 1 of 11 | | Inspiration | 10 | 6 (60%) | 6 of 10 | | Security and privacy | 3 | 1 (33%) | 0 of 3 | ## Which AI crawlers do websites block? Bytespider is blocked most, by 13 of 189 sites. GPTBot follows at 9 and ClaudeBot at 8. Sites treat bots differently: 7 of the 9 that block GPTBot do not block OAI-SearchBot, the bot behind ChatGPT search. Named in robots.txt, 189 sites we could read | Crawler | Company | Mainly for | Block it | Limit paths | Named | | --- | --- | --- | --- | --- | --- | | GPTBot | OpenAI | Training | 9 (5%) | 11 | 32 | | OAI-SearchBot | OpenAI | AI search | 2 (1%) | 11 | 20 | | ChatGPT-User | OpenAI | On request | 3 (2%) | 8 | 23 | | ClaudeBot | Anthropic | Training | 8 (4%) | 12 | 30 | | Claude-SearchBot | Anthropic | AI search | 4 (2%) | 5 | 11 | | Claude-User | Anthropic | On request | 3 (2%) | 4 | 10 | | PerplexityBot | Perplexity | AI search | 4 (2%) | 12 | 27 | | Perplexity-User | Perplexity | On request | 3 (2%) | 4 | 9 | | Google-Extended | Google | Training | 7 (4%) | 9 | 26 | | Applebot-Extended | Apple | Training | 6 (3%) | 5 | 16 | | Meta-ExternalAgent | Meta | Training | 7 (4%) | 6 | 16 | | Amazonbot | Amazon | Training | 7 (4%) | 7 | 21 | | CCBot | Common Crawl | Training | 9 (5%) | 7 | 20 | | Bytespider | ByteDance | Training | 13 (7%) | 3 | 18 | Good to know: Limit paths means rules for some areas, like login pages, with the rest open. Named counts every site with a rule for that bot. ## Who uses Content-Signal in robots.txt? 31 of 193 sites add a Content-Signal line, which says how they want their pages used. 22 say yes to AI training and 9 say no. 1 says no to AI answers, and 1 says no to search. ## How we measured On November 4, 2026 (UTC), we sent one GET for /robots.txt and one for /llms.txt to each brand's main address, four sites at a time, with a user agent that names Sitepeel. We followed redirects and did not retry. The 193 sites are every brand in Sitepeel's DESIGN.md library, which leans toward tech companies. This is not the whole web, and the shares would be lower there. Sitepeel is not in the sample. - llms.txt counts only as a real text file with a heading or links. A 404, or a redirect to an HTML page, counts as no file - 9 sites refused or failed the llms.txt request (cash.app, codepen.io, coolors.co, descript.com, fontsinuse.com, gitlab.com, openai.com, perplexity.ai, sonos.com). They count as not publishing, so 58% is a floor - 11 sites have no robots.txt, so they block nothing. 4 could not be read (cash.app, coolors.co, descript.com, sonos.com) and are left out - Blocks means a group names the bot and disallows an ordinary page. A few keep small exceptions, such as the home page - 2 sites disallow every crawler with a wildcard rule. They are not counted as blocking a named bot - Robots.txt is a request, not a lock. We did not test firewalls or what any bot really does - Google-Extended and Applebot-Extended are settings read from robots.txt, not separate crawlers ## Download the data and cite it The CSV has one row per brand: llms.txt result and size, robots.txt result, and the state of all 20 crawlers. Free to use with credit (CC BY 4.0). It reflects each file on the day we checked; sites change these often. Example: citation.md ``` Sitepeel (2026). Who lets AI read their site? llms.txt and AI crawler rules on 193 top websites. Checked November 2026. https://sitepeel.dev/reports/llms-txt-ai-crawlers-2026/ ``` - [Download the data (CSV, 193 rows)](https://sitepeel.dev/reports/llms-txt-ai-crawlers-2026/data.csv) - [All Sitepeel data reports](https://sitepeel.dev/reports/index.md) ## Questions and answers ### What percentage of websites have an llms.txt? 58% of the 193 top brand sites Sitepeel checked in November 2026 publish one. The sample leans toward tech companies, and we did not measure the whole web. ### How many websites block GPTBot? 9 of the 189 top brand sites we could read (5%) block GPTBot in robots.txt. 11 more limit it on some paths, and 12 name it and allow it. ### Which AI crawler is blocked the most? Bytespider, blocked in robots.txt by 13 of 189 sites. Among the bots from OpenAI and Anthropic, GPTBot is at 9 and ClaudeBot at 8. ### Does blocking GPTBot keep a site out of ChatGPT search? OpenAI uses separate bots: GPTBot for training and OAI-SearchBot for ChatGPT search, each with its own rule. 7 of the 9 sites that block GPTBot do not block OAI-SearchBot. ### What is an llms.txt file? A plain text file at /llms.txt that gives AI tools a short map of a site: a title, a summary and links to the pages that matter. It is a proposed convention, not a standard. ### Can I use this data? Yes. The CSV is free to use with credit (CC BY 4.0). Link to this page as the source. ## Further reading - [The llms.txt proposal](https://llmstxt.org/) - [Robots Exclusion Protocol (RFC 9309)](https://www.rfc-editor.org/rfc/rfc9309) - [OpenAI: overview of its crawlers](https://platform.openai.com/docs/bots) - [Content Signals](https://contentsignals.org/) ## Related pages - [What 191 top websites are built with](https://sitepeel.dev/reports/what-top-websites-are-built-with-2026/index.md): Sitepeel measured 191 top brand sites on November 4, 2026: 76% run React, 46% Next.js, 40% Tailwind CSS. Hosting and analytics too, free CSV. - [Fonts and colors of top websites](https://sitepeel.dev/reports/top-website-fonts-and-colors/index.md): The fonts and colors of 168 top websites, measured in October 2026: Inter on 26%, custom type on 54%, 44% in dark mode. Charts, method and a free CSV. - [What is DESIGN.md](https://sitepeel.dev/guides/what-is-design-md/index.md): DESIGN.md is a Markdown file that tells AI coding agents how your UI looks. See Google’s spec, three real examples, where to get templates and how to make one.