I’ve been blocking various “artificial intelligence” (“AI”) bots on my site. Why, you ask? Well, I don’t like the idea of my work being hoovered up to train “AI” data models; I don’t like that these companies assume my content’s available to them by default, and that I have to opt out of their scraping; I really don’t want anything I write to support these platforms, which I find unethical, extractive, deeply amoral, and profoundly anti-human. But! Sadly, this is the world we live in. So I’m opting out. There are many excellent tutorials out there on how to do this. (For my money, <a href="https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/">Neil Clarke</a> and <a href="https://coryd.dev/posts/2024/go-ahead-and-block-ai-web-crawlers/">Cory Dransfeldt</a> wrote two of my favorites, getting into not just the how of blocking “AI” bots, but also the why.) Most of those tutorials involve instructions on how to edit <a href="https://developer.mozilla.org/en-US/docs/Glossary/Robots.txt">your site’s <code>robots.txt</code> file</a>, providing it with a list of <dfn>user agents</dfn> — basically, the “names” of the bots, crawlers, and browsers visiting your site — and blocking them from accessing your website. But as I understand it, the one shortcoming of <code>robots.txt</code> is that it only works if visiting bots actually honor your <code>robots.txt</code> file. (<a href="https://developers.google.com/search/docs/crawling-indexing/robots/intro#understand-the-limitations-of-a-robots.txt-file">Google has a good intro on this</a>, if you’re interested.) That’s why I’ve opted to use <a href="https://developer.mozilla.org/en-US/docs/Learn/Server-side/ApacheConfigurationhtaccess">my site’s <code>.htaccess</code> file</a> to block these bots. As <a href="https://jasonsantamaria.com/">a friend</a> put it recently, <code>robots.txt</code> is a bit like asking bots to not visit my site; with <code>.htaccess</code>, you’re not asking. (If you’re wondering if robots that ignore <code>robots.txt</code> would perhaps lie about their user agent, you’re right to do so. Relying on user agent strings — which are basically <a href="https://bkardell.com/blog/UAGottaBeKidding.html">swamps filled with lies</a> — is incredibly <a href="https://css-tricks.com/browser-detection-is-bad/">fraught in any context</a>, including this one. That’s why I consider this approach to be marginally better than <code>robots.txt</code>, not a perfect solution.) Anyway. Here’s how I got things working. First, I polled a <a href="https://darkvisitors.com/">few</a> <a href="https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/">different</a> <a href="https://coryd.dev/posts/2024/go-ahead-and-block-ai-web-crawlers/">sources</a> to build a list of currently-known crawler names. Once I had them, I dropped them into <a href="https://developer.mozilla.org/en-US/docs/Learn/Server-side/ApacheConfigurationhtaccess#modrewriteandtherewriteenginedirectives">a <code>modrewrite</code> rule in my <code>.htaccess</code> file</a>: <pre><code class="language-apacheconf"><IfModule mod_rewrite.c> RewriteEngine on RewriteBase /

block “AI” bots

RewriteCond %{HTTPUSERAGENT} (AddSearchBot|AgentTimes|AI2Bot|AI2Bot-DeepResearchEval|Ai2Bot-Dolma|aiHitBot|AIWebIndex|amazon-kendra|amazon-QBusiness|Amazonbot|AmazonBuyForMe|Amzn-SearchBot|Amzn-User|Andibot|Anomura|anthropic-ai|ApifyBot|ApifyWebsiteContentCrawler|Applebot|Applebot-Extended|Aranet-SearchBot|atlassian-bot|Awario|AzureAI-SearchBot|bedrockbot|bigsur.ai|Bravebot|Brightbot|Brightbot 1.0|BuddyBot|Bytespider|CCBot|Channel3Bot|ChatGLM-Spider|ChatGPT Agent|ChatGPT-User|Claude-Code|Claude-SearchBot|Claude-User|Claude-Web|ClaudeBot|Cloudflare-AutoRAG|CloudVertexBot|Code|cohere-ai|cohere-training-data-crawler|Cotoyogi|CragCrawler|Crawl4AI|Crawlspace|Cursor|Datenbank Crawler|DeepSeekBot|Devin|Diffbot|DuckAssistBot|Echobot Bot|EchoboxBot|ExaBot|FacebookBot|facebookexternalhit|Factsetspyderbot|FirecrawlAgent|FriendlyCrawler|GeistHaus-PageFetcher|Gemini-Deep-Research|Google-Agent|Google-CloudVertexBot|Google-Extended|Google-Firebase|Google-Gemini-CLI|Google-NotebookLM|GoogleAgent-Mariner|GoogleAgent-URLContext|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|HenkBot|iAskBot|iaskspider|iaskspider/2.0|IbouBot|ICC-Crawler|ImagesiftBot|imageSpider|img2dataset|ISSCyberRiskCrawler|kagi-fetcher|Kangaroo Bot|Kimi-User|KlaviyoAIBot|KunatoCrawler|laion-huggingface-processor|LAIONDownloader|LCC|LinerBot|Linguee Bot|LinkupBot|Manus-User|meta-externalagent|Meta-ExternalAgent|meta-externalfetcher|Meta-ExternalFetcher|meta-webindexer|MistralAI-User|MistralAI-User/1.0|Mozilla-Tabstack|MyCentralAIScraperBot|NagetBot|netEstate Imprint Crawler|newsai|NotebookLM|NovaAct|OAI-SearchBot|omgili|omgilibot|OpenAI|opencode|Operator|PanguBot|Panscient|panscient.com|Perplexity-User|PerplexityBot|PetalBot|PhindBot|Poggio-Citations|Poseidon Research Crawler|QualifiedBot|Querit-SearchBot|QueritBot|QuillBot|quillbot.com|SBIntuitionsBot|Scrapy|SemrushBot-OCOB|SemrushBot-SWA|Shap-User|ShapBot|Sidetrade indexer bot|Spider|TavilyBot|Terra Cotta|TerraCotta|Thinkbot|TikTokSpider|Timpibot|TongyiBot|Trae|TwinAgent|UseAI|VelenPublicWebCrawler|WARDBot|Webzio-Extended|webzio-extended|wpbot|WRTNBot|YaK|YandexAdditional|YandexAdditionalBot|YiyanBot|YouBot|ZanistaBot) [NC] RewriteRule ^ – [F] </IfModule> </code></pre> If the crawler’s name appears anywhere in that gnarly-looking list of user agents, my site should block it. <details markdown="1"> <summary>If you’re interested, here’s how I automated this in my <del>Jekyll</del> Eleventy setup:</summary> While I could edit my <code>.htaccess</code> file every time a new <del>terrible</del> “AI” crawler hits the market, I thought it might be easier to use <a href="https://www.11ty.dev/docs/data/">data files</a> to build the whole thing dynamically. Here’s my current <code>bots.yml</code> file, which I’ve put in my site’s <code>data</code> directory: <pre><code class="language-yaml">- AddSearchBot

AgentTimes
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
AIWebIndex
amazon-kendra
amazon-QBusiness
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Amzn-User
Andibot
Anomura
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-Extended
Aranet-SearchBot
atlassian-bot
Awario
AzureAI-SearchBot
bedrockbot
bigsur.ai
Bravebot
Brightbot
Brightbot 1.0
BuddyBot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT Agent
ChatGPT-User
Claude-Code
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
CloudVertexBot
Code
cohere-ai
cohere-training-data-crawler
Cotoyogi
CragCrawler
Crawl4AI
Crawlspace
Cursor
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
ExaBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
GeistHaus-PageFetcher
Gemini-Deep-Research
Google-Agent
Google-CloudVertexBot
Google-Extended
Google-Firebase
Google-Gemini-CLI
Google-NotebookLM
GoogleAgent-Mariner
GoogleAgent-URLContext
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
HenkBot
iAskBot
iaskspider
iaskspider/2.0
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
img2dataset
ISSCyberRiskCrawler
kagi-fetcher
Kangaroo Bot
Kimi-User
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
LAIONDownloader
LCC
LinerBot
Linguee Bot
LinkupBot
Manus-User
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MistralAI-User/1.0
Mozilla-Tabstack
MyCentralAIScraperBot
NagetBot
netEstate Imprint Crawler
newsai
NotebookLM
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
opencode
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poggio-Citations
Poseidon Research Crawler
QualifiedBot
Querit-SearchBot
QueritBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Shap-User
ShapBot
Sidetrade indexer bot
Spider
TavilyBot
Terra Cotta
TerraCotta
Thinkbot
TikTokSpider
Timpibot
TongyiBot
Trae
TwinAgent
UseAI
VelenPublicWebCrawler
WARDBot
Webzio-Extended
webzio-extended
wpbot
WRTNBot
YaK
YandexAdditional
YandexAdditionalBot
YiyanBot
YouBot
ZanistaBot

</code></pre> With that file set up, <del>Jekyll</del> Eleventy can read its contents by accessing a <code>bots</code> variable. And in the template that generates my <code>.htaccess</code> file, I can use that variable to build my <code>modrewrite</code> rule: <pre><code class="language-liquid"> <IfModule modrewrite.c> RewriteEngine on RewriteBase /

block “AI” bots

RewriteCond %{HTTPUSERAGENT} ({{ bots | sort_natural | join: "|" }}) [NC] RewriteRule ^ – [F] </IfModule>

</code></pre> Whenever I add a new name to my <code>bots.yml</code> file, my <code>.htaccess</code> file will update itself automatically. Neat! </details> That’s how I’ve got things working, anyway. We’ll see how it works going forward. And as you might imagine, I really wish I didn’t have to think about any of this. <ins datetime="2024-5-12T16:16:13-5:00">Update: I’m grateful to <a href="https://techpolicy.social/@npdoty/112276122554383379">Nick Doty</a> for pointing out that <code>Google-Extended</code> doesn’t appear in a user agent string, which means it can only be blocked by <code>robots.txt</code> files. So for now, I’m duplicating the above logic in both my <code>.htaccess</code> file and in <a href="https://ethanmarcotte.com/robots.txt">my <code>robots.txt</code></a>. I should’ve known it wouldn’t be quite that easy! Capitalism is working fine! Computers were a mistake!</ins> <hr /> Note: I apologize, but I can’t provide any technical support for anything outlined in this post. With that said, all of the code blocks on this page are built dynamically, using the data file I use to build my <code>.htaccess</code> file. As long as I don’t break anything on this little site — which is, to be clear, quite likely! — the snippets above should stay current whenever I add a new bot to the list.

<hr /> This has been “<a href="https://ethanmarcotte.com/wrote/blockin-bots/">Blockin’ bots.</a>” a post from <a href="https://ethanmarcotte.com/wrote/">Ethan’s journal.</a> <a href="mailto:listener+rss@ethanmarcotte.com?subject=Reply%20to:%20“Blockin’ bots.”">Reply via email</a>