Defending Your Site from AI-Training Crawlers — robots.txt, noai & CDN Blocks (US/EU)

Written: 2026-05 / Jurisdiction focus: United States & EU (Japan is covered in the Japanese edition) / last_updated: 2026-05-31 Changelog: 2026-05-31 first published as an international (US/EU) edition

Scope of this article: it is about defending your own site from AI-training crawlers. It is the mirror image of the legal boundaries of scraping: same technology (robots.txt◆ / RFC 9309), opposite vantage point — that one is about the fetcher's duties, this one about the defender's signals and technical blocks.

About this series: part of YATA-NODE's "rights basics" series. The hub is the pre-ship rights checklist; related pieces are the legal boundaries of scraping (the reverse vantage point), how to read AI service terms (the consumer-vs-enterprise-API opt-out axis), and AI-generated content copyright (which covers the EU AI Act◆ Art. 50 disclosure duty and copyrightability). A reverse layer — protecting the private data you entrust to US providers from government access (CLOUD Act, FISA 702) — is in handing your data to US providers (this article defends public content; that one covers the sovereignty of data you hand over). The companion Japanese edition covers the same ground under Japanese law. Each piece stands on its own.

A few terms (for newcomers):

• Crawler / bot: a program that automatically visits web pages to fetch their contents; it identifies itself with a User-Agent string.
• robots.txt: a file at a site's root (/robots.txt) telling crawlers, per User-Agent, which pages they may fetch (standardized as RFC 9309).
• X-Robots-Tag: an HTTP response header carrying robots directives — usable for non-HTML files (PDFs, images) too.
• TDM (text and data mining): automated analysis of large volumes of text/data, the category EU law uses for most "training data" questions.
• opt-out: a declaration refusing a particular use; in the EU, a machine-readable opt-out is what reserves rights against the DSM Art. 4 TDM exception.

"Will my site's content be used for AI training, and is that legal?" — the answer depends on jurisdiction, and on a defense you can act on today. This article is about defending your own site; the legal boundaries of scraping covered the fetching side.

Telling the measures apart — "stop," "signal," and "record"

Before the details, the five measures here differ in kind. Treating them as one "AI countermeasure" hides the point: some block technically, some signal intent, some serve as evidence later.

Of these, the only ones that actually stop a crawler are CDN/WAF-level blocks (Cloudflare, and equivalent AI-bot blocking on Vercel / Netlify); the rest signal and record. Even technical blocks aren't all-powerful — they mainly catch known, detectable crawlers (see below).

Where AI-training exceptions land — US and EU

"Is it illegal for AI to train on my site's content?" differs by jurisdiction. (The merits of individual cases come in the next section.)

United States — fair use (17 U.S.C. §107)

Fair use weighs four factors (purpose, nature, amount, market effect) in a fact-specific, case-by-case analysis; AI training is not automatically fair use. In practice the outcomes diverge: Thomson Reuters v. Ross (Feb 2025) rejected fair use, while Bartz v. Anthropic and Kadrey v. Meta (both June 2025) held training to be fair use. The result depends on the case.

European Union — DSM Directive 2019/790, Arts. 3 & 4

Article 4 permits TDM unless the rightsholder reserved the use by machine-readable means (a real opt-out); Article 3 covers scientific research by eligible institutions and can't be overridden by an opt-out. So to say "don't train on this," you need a machine-readable opt-out — and the robots.txt / header signals in this article's defense section are a strong candidate for it. Whether they suffice under Art. 4 depends on each member state's implementing law and how strictly it reads the machine-readable requirement (some read it strictly; it is not uniform) — confirm against the primary text and a professional. Separately, the EU AI Act's general-purpose AI (GPAI) provider obligations took effect 2025-08-02 (these bind GPAI model providers, not site operators generally), with the European Commission's enforcement powers from 2026-08-02 (the duty and the regulator's enforcement start on different dates). In the US, state law is moving too: California AB 2013 took effect 2026-01-01, requiring training-data transparency from developers who publicly deploy generative-AI systems in California (again, not a duty on site operators generally). A regulatory calendar of "what applies, and when" helps you plan your own site and products.

IETF AIPREF WG (under discussion)

The IETF AIPREF WG was chartered in 2025-04 and is discussing preference signaling for AI crawlers (Internet-Draft stage, pre-standardization as of 2026-05) — a signal alongside or reinforcing robots.txt. Because it is pre-standard, this article treats it as under discussion and makes no firm claim.

Note: this article does not cover the EU AI Act Art. 50 disclosure duty (how AI-generated output is labeled, expected to apply 2026-08-02) or the copyrightability of AI output — both are covered in AI-generated content copyright. "How content was used for training" (this article) is a different layer.

Litigation — what's been decided, and what hasn't

Through 2025–2026, AI-training suits surged — various trackers count more than 100 pending in the US alone (counts vary by definition, so treat "100-plus" as an order-of-magnitude figure, not a fixed number). But the published merits rulings and settlements are few (roughly the Thomson Reuters first-instance ruling, the Getty UK first-instance ruling, and the Bartz SJ plus settlement — several on appeal, settled, or at the SJ stage, where a higher court could shift the scope; not necessarily final judgments). Most are pending or on appeal, with some SJ dates pushed to 2027. The key to reading them is to separate, per case, what was decided from what wasn't — and to read "pending / hearing on [date]" rather than "the court ruled."

Thomson Reuters v. Ross Intelligence (D. Del., 2025-02-11)

Reported as a merits ruling rejecting fair use for a non-generative AI's training on Westlaw headnotes. The ruling is expressly limited to non-generative AI, Ross is on appeal, and the reach to generative AI is unresolved — one case does not mean "fair use is gone."

Bartz v. Anthropic (N.D. Cal., 2025-06-23 SJ → 2025-09-25 settlement)

Judge Alsup's summary judgment held that using lawfully acquired books to train was fair use (transformative), while acquiring and retaining pirated copies was not — "training transformativeness" and "lawful acquisition" are separate layers. The parties then settled for about $1.5 billion (a post-SJ money settlement). Preliminary approval was granted on 2025-09-25, and a final approval (fairness) hearing was held on 2026-05-14; no final approval order issued at the hearing, the matter remained under further briefing, and final approval is not yet confirmed as of June 2026. Note that a class-action settlement becomes final only through court approval (preliminary / final), so the approval stage and final date are not yet settled; per party explanations, the settlement mainly releases past acquisition, retention, and use, leaving future training licenses and output-based claims separate. This case did not hold "training itself is illegal," and a settlement is not "losing on the merits."

Other cases (pending / other jurisdictions), in brief

Primaries in References:

• NYT v. OpenAI (S.D.N.Y.; consolidated into MDL 1:25-md-03143 on 2025-04-03): a 2025-04-04 partial MTD ruling let the core copyright claims proceed (not a merits ruling; pending).
• Kadrey v. Meta (N.D. Cal., 2025-06-25 SJ): Meta won, but the court expressly said it was because plaintiffs failed to prove market dilution — implying a better-argued case could differ (per the summary-judgment order; see References).
• Getty Images v. Stability AI ([2025] EWHC 2863 (Ch), UK High Court, 2025-11-04): Getty dropped its primary copyright and database-right claims (training occurred outside the UK), and the court separately rejected the secondary-copyright claim on the merits — holding the model weights store no copy of the works ("no copies in the model"); only a narrow trademark claim partly succeeded (UK law, CDPA 1988 — not EU law).
• Andersen v. Stability AI (N.D. Cal., 2024-08-12): DMCA §1202 was dismissed with prejudice, but the direct and induced copyright claims survived and discovery is ongoing (no merits SJ / settlement confirmed as of 2026-05).
• GEMA v. OpenAI (Munich Regional Court I, 2025-11-11): on press and GEMA's own announcement, reported as a significant European ruling on an LLM's memorization and regeneration of song lyrics (reportedly denying the German TDM exception; OpenAI is said to have appealed). The dispute centers on memorization/regeneration, not "AI training in general" — all via party announcements and secondary reporting; the primary judgment and appeal docket are not yet publicly available. Notably, a different German court (Hamburg) found for LAION (applying research / general TDM exceptions, now on further appeal to the Federal Court of Justice) — so outcomes split even within the EU (the Netherlands, for instance, reads the machine-readable opt-out requirement strictly). These reflect differences in each system's rules and facts, not a ranking.

What emerges is that neither "training is free" nor "fair use is dead" generalizes. Fair use is fact-specific, and whether your site's content becomes a problem turns on acquisition path, use, and market effect. So the next section is about what you can do now — signaling intent and blocking technically — rather than waiting for outcomes.

Defending your own site — combine "signal" and "stop"

The legal boundaries of scraping covered the line a fetcher must honor. This is the reverse: using the same RFC 9309 (robots.txt) from the defender's side. We turn the table above into steps.

robots.txt (RFC 9309) — signal intent and leave a record

Standardized in 2022; crawlers aren't technically obliged to obey, but it functions as evidence of your intent in a dispute (legal force is limited). Treat it as "declare and record," not "stop." Name the User-Agents you want to exclude — common AI crawlers include GPTBot (OpenAI training), ChatGPT-User (conversational fetch), CCBot (Common Crawl), ClaudeBot (Anthropic), Google-Extended (Google AI training), and PerplexityBot (per each vendor's official docs, as of 2026-05). Note Google-Extended is a robots.txt control product token, not a separate crawler User-Agent — you list it in robots.txt, but it won't show up as its own User-Agent in access logs; it is also distinct from search's Googlebot, so blocking it doesn't affect normal search indexing.

But there's a hard limit: whether robots.txt is honored is up to the crawler, and 2025–2026 is polarized. Some vendors document robots.txt / User-Agent controls (OpenAI, Anthropic, Google, Meta, Mistral, Apple, per their docs), while xAI (Grok) and Perplexity are reported to evade robots.txt control via IP rotation or User-Agent spoofing (Cloudflare revoked Perplexity's verified-bot status in 2025-08), and DeepSeek does not publish an official User-Agent (these reflect operational and business choices, not a ranking). A large-scale empirical study (preprint, 2025) likewise reported that AI search crawlers rarely consult robots.txt. So "write it in robots.txt and you're protected" is only half true — it works as intent and evidence, but reliable blocking needs the technical block below plus machine-readable metadata and bot verification, layered.

X-Robots-Tag (HTTP header) — reach static files too

Returns robots directives via an HTTP header, useful for non-HTML files (PDFs, images). If robots.txt is the site-wide rule, X-Robots-Tag carries the same intent at the HTTP layer, per file — worth combining on sites that host PDFs or figures.

noai / noimageai meta tags — an extra HTML-level signal

Implemented as defensive depth, but they carry no legal force and are not a de facto standard as of 2026-05 (platform-specific at best). "No harm in adding, but don't rely on it alone."

Cloudflare AI-crawler block (one-click) — the layer that actually stops

The three above all signal. What technically blocks is a CDN / WAF feature. Cloudflare began offering one-click AI-crawler blocking on all plans (including free) in mid-2024, and in 2025-07 changed the default for new domains to "block AI." Unlike a signal, it works even on crawlers that ignore robots.txt — but it isn't all-powerful: it mainly catches known, detectable crawlers, not necessarily disguised, unidentified, or routed-around ones, and doesn't reach paths that bypass Cloudflare. Cloudflare's own figures put most AI-bot traffic at training purposes, with the share ignoring robots.txt rising — hence the layered defense.

Cloudflare pay-per-crawl (under development)

A private-beta proposal (as of 2026-05) for charging / licensing AI crawlers — potentially a third option beyond "block or allow." This article records it as a development.

Where to put it, by host (mainly for company sites)

The robots.txt content is the same, but placement and headers differ. Disallow is a robots.txt syntax keyword — don't translate it; the comment after # explains it.

# Intent signal for AI training crawlers (example; search's Googlebot is not targeted)
# "Disallow" is a robots.txt syntax keyword (do not fetch).
User-agent: GPTBot
Disallow: /          # do not fetch (whole site)

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

In every case, stack it in the order of the table: signal with robots.txt → extend to static files with X-Robots-Tag if needed → block with Cloudflare when you must truly stop. After deploying, review per-User-Agent access logs periodically to see whether your signal is being ignored (your cue to switch to blocking) — noting that robots.txt-only tokens like Google-Extended won't appear as a distinct User-Agent.

A first step for solo / side-project developers

This can look heavy, but for a personal blog or GitHub Pages, the first step is just a few Disallow lines in robots.txt. Copy the example into your publish directory (the repo's public root for GitHub Pages, often public/ for static sites) as robots.txt, and the signal is in place. Don't aim for perfection — start by recording the intent not to be used.

Summary — a defense checklist by role

You don't need to do everything. Use this as a map to decide, for your role, how far to signal and where to start blocking.

[If you run a company website]
□ 1. Aggregated one month of per-User-Agent access to your main site
□ 2. Added Disallow for GPTBot / CCBot / ClaudeBot / Google-Extended / PerplexityBot to robots.txt
□ 3. Considered extending intent to static files (PDFs / images) via X-Robots-Tag
□ 4. Decided whether to add noai / noimageai, understanding it has no legal force
□ 5. Decided whether to enable Cloudflare's one-click block for anything you must truly stop
□ 6. Recorded the robots.txt date and contents as evidence of your intent
□ 7. Set up a route to confirm any final dataset-license question with legal / IP counsel

[If you keep public repos / blogs as a solo or side-project dev]
□ 1. Checked whether your blog / GitHub Pages has a robots.txt
□ 2. Added Disallow for the AI crawlers you want to exclude
□ 3. Placed robots.txt in the right spot for your host (public root / public/)
□ 4. (Optional) Added a noai meta tag, understanding it's a weak extra signal

The axis is simple: robots.txt, X-Robots-Tag, and noai signal intent and leave a record; CDN-level blocks actually stop (mainly known) crawlers — they work even on those that ignore robots.txt, but disguise and rerouting aren't fully stopped, so layering is the premise. The litigation in this article is too fluid to call "training is free" or "fair use is dead" — so rather than waiting for outcomes, get your signals and records in order now.

Opt-out as an AI-service user (the default differences between consumer plans and enterprise APIs) is covered in the terms-of-service post; the copyrightability and labeling of AI output (EU AI Act Art. 50) come in AI-generated content copyright. This article stays on defending your own site.

References

Primary materials where available; for some fast-moving AI cases, reputable analyses and dockets are used to locate the primaries. A trailing (primary) marks a primary source; (supporting) marks commentary / a docket used to locate primaries. All sources reflect verification as of 2026-05; where a point rests on secondary reporting, the citation is tagged (supporting).

Statutes, regulations & standards

• 17 U.S.C. §107 — Fair use(primary)
• Directive (EU) 2019/790 (DSM), Arts. 3–4 — TDM(primary)
• EU AI Act — application timeline (GPAI 2025-08; enforcement 2026-08-02)(primary)
• GPAI Code of Practice (final, 2025-07-10)(primary)
• IETF RFC 9309 — Robots Exclusion Protocol(primary)
• IETF AIPREF Working Group(primary — under discussion)
• IETF draft-ietf-aipref-vocab(primary — Internet-Draft)
• Google Search Central — Robots meta tag / X-Robots-Tag(primary)
• CAWG — Training and Data Mining Assertion (cawg.training-mining)(primary)

Case law

• NYT v. OpenAI (S.D.N.Y. MTD opinion, 2025-04-04)(primary)
• In re: OpenAI Copyright Infringement Litigation (MDL docket, S.D.N.Y.)(supporting — docket)
• Getty Images v. Stability AI ([2025] EWHC 2863 (Ch))(primary — UK High Court)
• Kadrey v. Meta Platforms (N.D. Cal. 3:23-cv-03417, Doc. 598)(primary)
• Andersen v. Stability AI (N.D. Cal., redacted order)(primary)
• Thomson Reuters v. Ross Intelligence (D. Del. opinion, 2025-02-11)(primary — court opinion); analysis: Jenner & Block(supporting)
• Bartz v. Anthropic — fair-use SJ order (Doc. 231)(primary) / docket 3:24-cv-05417 (CourtListener)(supporting — docket) / $1.5B settlement explainer (Authors Guild)(supporting)
• GEMA v. OpenAI — announcement (GEMA)(supporting)

Crawler tech & host-level defense

• Cloudflare — Block AI bots, scrapers and crawlers with a single click(primary)
• Cloudflare — Content Independence Day (new-domain default AI block, 2025-07-01)(primary)
• Cloudflare — Perplexity is using stealth, undeclared crawlers (2025-08-04)(primary)
• Cloudflare — Introducing pay-per-crawl(primary — private beta, under development)
• Anthropic — Does Anthropic crawl data from the web (ClaudeBot)(primary)
• Vercel — AI bots managed ruleset + Web Bot Authentication(primary)
• Netlify documentation (User Agent Blocker extension)(primary)
• arXiv 2505.21733 — AI search crawlers' robots.txt compliance study(supporting — academic)

On AI assistance: Starting from points the author already knew, the author used LLMs (Claude by Anthropic, with a second, independent model cross-checking the legal facts) to research, organize, and summarize, then verified the facts against the primary sources cited above and revised accordingly. AI-training copyright is fluid, so statutes, effective dates, and the reach of rulings reflect verification as of 2026-05; points resting on secondary reporting are tagged (supporting) in the references. For the "not legal advice" note and what is out of scope, see the disclaimer at the top.

About the author

More than 20 years of electrical and software development — from control engineering at a major electronics manufacturer — plus about 10 years of solo development. Across hardware and software, and across enterprise and individual work, I publish the basics that "become a risk if you don't know them," and I plan to cover practical ways to use AI as well. More at About this blog.