The Legal Boundaries of Web Scraping for US/EU Builders — CFAA, GDPR, DSM & AI Training

Written: 2026-05 / Jurisdictions in scope: United States & EU (Japan is covered in the Japanese edition) / last_updated: 2026-05-25 Changelog: 2026-05-25 first published as an international (US/EU) edition

About this series: this is the international (US/EU) edition of the third post in YATA-NODE's "rights basics" series, after the pre-ship rights check and reading OSS licenses; the companion Japanese edition covers the same ground under Japanese law. The licensing of the assets themselves (images, audio, fonts) is the 4th post.

A few terms (for newcomers):

• Scraping / crawling: fetching data from web pages automatically with a program (strictly, crawling = visiting pages, scraping = extracting; treated together in practice).
• robots.txt: a file by which a site tells automated crawlers which pages they may fetch (standardized as RFC 9309).
• Terms of service (ToS): the conditions a site sets for use; depending on how you accept them, they can form a contract.
• TDM (text and data mining): automated analysis of large volumes of text/data, including AI training.
• CFAA: the US Computer Fraud and Abuse Act (18 U.S.C. §1030), which prohibits accessing a computer "without authorization."

"Is my scraping legal?" — there's no one-size answer. And scraping is no longer something only data-collection firms do. The moment you hand an AI a URL and say "look this up" or "summarize this," that AI service's fetcher retrieves the page on your behalf. Crawlers are split by purpose (user-initiated fetch such as ChatGPT-User / Perplexity-User vs. training crawls such as GPTBot), and "look it up" drives the former. (Note: Google-Extended is not a standalone HTTP crawler but a robots.txt◆ token that controls whether your site is used for training, e.g. for Gemini.) This applies when the AI actually fetches a page in real time; if it answers from its trained knowledge, no retrieval happens. But the moment you make it fetch a page, "letting AI look it up" is scraping too — and you are the party who triggered it (where legal responsibility lands is a separate, unsettled question — see below).

This isn't hypothetical. In July 2023, OpenAI suspended ChatGPT's "Browse with Bing" feature after users found that, given a URL, it could reproduce the full text of paywalled articles; OpenAI pulled it "to do right by content owners" (it returned later, after fixes). The issue wasn't breaking a login wall — it was that fetching a page on your behalf surfaced content the publisher had gated. A casual "look this up" sits right next to how the fetched content gets used. The backbone here is US and EU law; for Japan, see the Japanese edition.

Four legal lenses that cross over scraping

Before you use what you scraped, sort your case into four legal lenses. None decides the question alone — you check how your case lands in each.

A fifth, cross-cutting risk in the US is trespass to chattels — not about data rights, but about burdening someone's servers. eBay v. Bidder's Edge (N.D. Cal. 2000) first applied it to excessive automated access; Intel v. Hamidi (Cal. 2003) then required actual impairment, raising the bar (more on "how much is too much" below).

The point: the four lenses aren't "trip one and you're out." A use can be fine on copyright yet fail on personal data; you may clear the auth wall yet still breach a contract. Next, the three things that get conflated most — robots.txt, terms, and the auth wall.

robots.txt, terms, and the auth wall — different in kind, different in risk

Three "looks-prohibited" things meet you early: robots.txt, the terms of service, and the access barriers (an authentication login, plus CAPTCHA / IP limits). They look alike but are different in legal kind, so the risk of crossing each differs.

① robots.txt = a technical norm. Standardized as RFC 9309, which itself states it is "not ... a form of access authorization." Google likewise says it is up to the crawler to obey. So robots.txt doesn't legally authorize or forbid access by itself — but ignoring it and then hammering a site can become an unfavorable fact under other doctrines (trespass to chattels in the US; the EU DSM◆ opt-out below). Page-level signals exist too: <meta name="robots"> in the <head> and the X-Robots-Tag HTTP header (which also covers PDFs / images), including noai / noimageai for AI opt-out — not yet an industry standard or law in 2026, though in the EU they may be argued as a machine-readable reservation under DSM Art. 4 (whether such a reservation is valid is unsettled).

② Terms of service = a contract. US courts have drawn a sharp line for public scraping: in Meta v. Bright Data (N.D. Cal. 2024), Judge Chen held that Meta's ToS bind logged-in account holders, not a third party scraping while logged out — a non-party isn't bound by the contract. (Whether a "clickwrap" vs. "browsewrap" acceptance forms a binding contract is its own US line of cases — Specht v. Netscape being the classic.) The EU strikes a different balance: in Ryanair v. PR Aviation (CJEU, C-30/14), for a database not protected by copyright or the sui generis right, national contract law can still enforce a ToS restriction.

③ The authentication wall is the CFAA's core territory; CAPTCHA / IP limits are murkier. Crossing a login gate is the paradigm "gate down" case — closest to serious civil/criminal exposure. After Van Buren v. United States (2021), the CFAA's "without authorization" is read narrowly via a "gates-up-or-down" test, and hiQ v. LinkedIn (9th Cir. 2022) applied that to hold that scraping public (non-gated) data isn't a CFAA violation. But CAPTCHAs and IP / rate limits are different: they screen bots and load, not identity, so whether defeating them is "unauthorized access" under the CFAA is unsettled — and the more concrete risks are trespass to chattels and breach of contract. (Separately, circumventing a technical protection measure on copyrighted content — encryption, geo-locks — is its own matter under DMCA §1201, not a plain login bypass.)

In short: robots.txt is a norm (evidence under other laws), terms are a contract (they bind you mainly when you are a party), and the authentication wall is the CFAA's core domain (while CAPTCHA / IP limits are more contested). Tell which one you are crossing first.

"How much access is too much?" — there is no fixed number

We have used "excessive access" a few times. How much is too much? There is no fixed numeric threshold. In the US, the relevant doctrine — trespass to chattels — turns not on requests-per-second but on whether you caused actual impairment to the server. eBay v. Bidder's Edge (2000) enjoined a bot that continuously hit eBay's servers; but Intel v. Hamidi (2003) held that without actual harm or impairment there is no trespass, raising the bar. So "excessive" is measured by impact on the target, not an absolute rate.

That cuts both ways: a modest rate can still cause problems if the target is fragile, and a robust site may be unbothered by the same rate. And the stakes aren't only civil. In Japan, a developer crawling a library catalog at roughly one request per second — a rate the technical community called "polite" — was arrested for suspected business obstruction when the library's own buggy system failed, and held in custody for about 20 days before the case was dropped (the Japanese edition tells that story in full). The lesson travels: you can get caught up even when you intended nothing excessive.

So the real defense is to make your good faith visible:

• Put a name on your program (a contactable User-Agent). The User-Agent string your program sends is freely settable; put your contact (a URL or email) in it — e.g., MyResearchBot/1.0 (+https://example.com/about; [email protected]). Then an operator who sees it in the logs can reach you before escalating.
• Start slow (well under a few requests per second; scale to the target's size).
• Honor the server's "wait" signals (back off on HTTP 429 / Retry-After).
• Limit concurrency, and prefer off-peak hours.
• Keep records (URLs, timestamps, rate) so you can later show there was no malice.

With no fixed safe number, watch the target's responses (latency, error rate) and adjust. Staying in a state where you can credibly say "I wasn't burdening anyone" is the best insurance.

Copyright and AI training — fair use is being fought right now

If you scrape to train or fine-tune a model, US copyright turns on fair use (17 U.S.C. §107) — there is no statutory TDM exception. And the case law is actively split (status as of 2026-05):

• Thomson Reuters v. Ross Intelligence (D. Del., Feb 2025): the first ruling to reject fair use for AI training — Ross copied Westlaw headnotes to build a competing legal-research tool, and the court found market substitution. Expressly limited to non-generative AI, leaving generative AI open; on appeal.
• Bartz v. Anthropic (N.D. Cal., June 2025): training on lawfully acquired books was held to be fair use (transformative), but acquiring and retaining pirated copies was not — and the parties settled for about $1.5 billion. The lesson: "training transformativeness" and "lawfulness of how the data was acquired" are separate layers, and the latter is the big-dollar risk.
• Kadrey v. Meta (N.D. Cal., June 2025): training Llama was held "highly transformative," but the court stressed Meta won because the plaintiffs failed to prove market dilution — implying a better-argued case could come out differently.

The takeaway isn't "training is fine" or "training infringes" — it's that fair use is fact-specific and unsettled, so records and conservative defaults matter.

In the EU, training data runs mostly through the DSM Directive's TDM exceptions (2019/790). Article 4 permits TDM with lawful access unless the rightsholder reserved the use by machine-readable means (an opt-out with teeth); Article 3 covers scientific research by eligible institutions and can't be overridden by an opt-out. This is why robots.txt-style signals and ai.txt / AIPREF proposals matter in the EU — and the EU AI Act◆ (Regulation (EU) 2024/1689; its Article 50 transparency duties — such as marking synthetic content in machine-readable form and disclosing deep fakes — start to apply on 2 Aug 2026) layers disclosure duties on general-purpose AI providers on top.

When personal data is involved, the calculus changes

If what you scrape includes personal data, a different body of law moves to the front — and the catch is that "publicly available" does not mean "free to use."

In the EU, the GDPR◆ governs collection and use. Processing needs a lawful basis (Art. 6), and "it was public" or "we have a legitimate interest" is not, by itself, enough — multiple authorities found exactly that against Clearview AI, which scraped web images at scale to build a face-recognition database and was fined €20 million by Italy's Garante (with deletion and a processing ban; Greece and France issued comparable fines). The UK ICO's penalty took a more contested path — overturned by a tribunal in 2023, then the ICO's 2025 appeal succeeded (Upper Tribunal, [2025] UKUT 319 (AAC)) and the case was remitted to the First-tier Tribunal; Clearview was then granted permission to appeal that ruling to the Court of Appeal (Dec 2025), so it remains contested as of 2026-05. The GDPR also reaches controllers outside the EU. And when you collect personal data indirectly (from third-party pages rather than from the person), the controller generally owes the data subject notice under Art. 14 — a burden that lands on you.

In the US, there is no single federal privacy law; instead a state patchwork (California's CCPA / CPRA and a growing list of others) governs collecting and "selling / sharing" personal information, with consumer rights to access and delete. Scope and definitions differ by state, so "US-wide" answers are rare.

A note for engineers doing this at work: business-scale collection is where this bites hardest. Before you start, confirm ① whether the target includes personal data (customer lists, support histories, profiles on social sites), ② whether you can state your purpose for collecting and using it (under the GDPR, indirect collection triggers an Art. 14 notice duty), and ③ that you won't link records or profile people (combining scattered data into one identity; automatically inferring interests, behavior, or creditworthiness). "We just gathered public info" is not a defense here.

Three misreadings to clear

Three traps, each with the other side of the coin. All show it is not a simple legal / illegal binary.

Misreading ① "If robots.txt has Disallow, fetching it is instantly illegal." robots.txt is a norm, not a law; ignoring it isn't itself a crime, and (with no auth gate) it isn't a CFAA violation either. But "so I can ignore it" is also wrong — ignoring it plus heavy access can count against you under trespass to chattels, and in the EU, disregarding a valid machine-readable reservation can take you outside the DSM Art. 4 TDM exception. robots.txt is neither "instant arrest" nor "pure decoration" — it is evidence that bites in combination with other law.

Misreading ② "If a page is public, I can take and use it freely." In the US, scraping non-gated public data isn't a CFAA violation (hiQ v. LinkedIn). But "not a CFAA violation" ≠ "anything goes": copyright, personal data (Clearview), and the terms of service remain separate layers. "Public = free" is the most expensive misreading.

Misreading ③ "If the terms say 'no scraping,' doing it is a crime." A ToS breach is first a contract matter (civil), not a crime. And a party who never agreed — a logged-off third party — is hard to bind (Meta v. Bright Data). Conversely, once you create an account and click "I agree," you are bound, and scraping after authenticating breaches the contract more readily. The EU (Ryanair) enforces contractual restrictions more broadly. So it is neither "terms = crime" nor "terms are void" — it turns on whether you became a contracting party.

The common thread: "it says forbidden somewhere" or "it's public" doesn't settle it. You have to identify the law that applies and your own position (party to a contract? crossed an auth gate? handling personal data?).

A five-axis checklist before you use the data

Finally, the five axes to check your case against before you use what you fetched. The same five apply whether you scrape by hand or have an AI do the web research.

□ 1. robots.txt: is the target URL disallowed? Did you honor noindex / noai?
□ 2. Terms of service: any no-scraping clause? Are you logged in (= a contracting party) or a logged-off third party?
□ 3. Access barriers: did you bypass an authentication login (= CFAA risk)? Avoid circumventing CAPTCHA / IP limits too (contested under the CFAA; clearer trespass / contract risk); circumventing a technical protection on copyrighted content adds DMCA §1201
□ 4. Personal data: does the target include it? If so, can you state a lawful basis and purpose (GDPR Art. 6 / Art. 14; US state laws)?
□ 5. Purpose: plain analysis / AI training (fair use is unsettled; EU Art. 4 opt-out) / redistribution (copyright + DB rights)?
□ +. Rate: contactable User-Agent, start slow, honor 429 / Retry-After (avoid trespass-to-chattels harm)
□ +. Records: keep source URLs, timestamps, and rate for accountability

A note by reader:

• If you publish or monetize a side project: redistributing what you collected puts axis 5 (copyright) front and center. Facts you learned are free; someone else's expression copied verbatim is not.
• Solo developers: when you tell an AI agent "go research this site," you are still the party — axes 1–5 apply, with special care on axis 3 (don't instruct it to bypass authentication).
• Engineers at work: business-scale collection raises axis 4 (personal data) and axis 2 (you are easily a contracting party). Internal rules and access records pay off.

Summary — not "is it legal?" but "which lens, and where is the line?"

The legality of scraping isn't a single yes/no. The work is to sort your case into four legal lenses — computer-access law (CFAA), copyright / TDM (fair use; EU DSM Art. 4), personal data (GDPR; US state laws), and contract (ToS◆) — plus how you access, and to find the line in each. robots.txt is a norm (evidence under other laws), terms are a contract (it depends on whether you are a party), and the authentication wall is the CFAA's core domain (CAPTCHA / IP limits are more contested) — different in kind, different in risk. And "it's public" or "it says forbidden somewhere" doesn't settle it.

When you are near the edges — crossing an authentication gate, collecting personal data at scale, or substituting for a paid dataset — confirm with a qualified attorney or your legal team before you start. This article is only a first map to judge whether you need that conversation.

And don't read all this and freeze. The risk is in charging ahead without knowing; once you have the lenses, you can use scraping and AI research without undue fear. Knowing the boundaries isn't about flinching — it is about moving forward with confidence.

Related: The Pre-Ship Rights Checklist (rights overview) / Reading OSS Licenses Before You Use Them (adopting others' code). The licensing of the assets themselves (images, audio, fonts) is covered in licensing external assets, and the copyrightability of AI-generated work in copyright in AI-generated work.

References

Primary materials where available; for some fast-moving AI cases, reputable analyses and dockets are used to locate the primaries. A trailing (primary) marks a primary source; (supporting) marks commentary / a docket used to locate primaries.

Statutes & regulations

• 18 U.S.C. §1030 — Computer Fraud and Abuse Act(primary)
• 17 U.S.C. §107 — Fair use(primary)
• 17 U.S.C. §1201 — Circumvention of technological protection measures(primary)
• Directive (EU) 2019/790 (DSM), Arts. 3–4 — TDM(primary)
• Regulation (EU) 2016/679 (GDPR)(primary)
• Regulation (EU) 2022/2065 (Digital Services Act)(primary)

Standards & guidance

• IETF RFC 9309 — Robots Exclusion Protocol(primary)
• Google Search Central — Intro to robots.txt(primary)
• OpenAI — Overview of OpenAI crawlers (GPTBot / ChatGPT-User)(primary)

Case law

• Van Buren v. United States, 593 U.S. 374 (2021)(primary)
• hiQ Labs v. LinkedIn (9th Cir. 2022, No. 17-16783)(primary)
• Meta v. Bright Data (N.D. Cal. 3:23-cv-00077, docket)(supporting)
• eBay v. Bidder's Edge, 100 F. Supp. 2d 1058 (N.D. Cal. 2000)(primary)
• Ryanair v. PR Aviation (CJEU, C-30/14)(primary)
• Thomson Reuters v. Ross Intelligence (D. Del. opinion, 2025-02-11)(primary — court opinion); analysis: Jenner & Block(supporting)
• Bartz v. Anthropic — fair-use SJ order (Doc. 231)(supporting — order PDF) / docket 3:24-cv-05417 (CourtListener)(supporting) / $1.5B settlement explainer (Authors Guild)(supporting)
• Kadrey v. Meta Platforms (N.D. Cal. 3:23-cv-03417, Doc. 598)(primary)

Penalties & events

• Clearview AI — Italian Garante €20M (EDPB)(primary)
• ICO v. Clearview AI — UK Upper Tribunal, [2025] UKUT 319 (AAC)(primary — UK Upper Tribunal; remitted to FTT, Clearview granted permission to appeal to the Court of Appeal Dec 2025)
• ChatGPT "Browse with Bing" suspension (2023-07, SiliconANGLE)(supporting)

On AI assistance: Starting from points the author already knew, the author used LLMs (Claude by Anthropic, with a second, independent model cross-checking the legal facts) to research, organize, and summarize, then verified the facts against the primary sources cited above and revised accordingly. Some points here — the liability of a user who has an AI fetch on their behalf, and whether fair use covers AI training — have little settled case law, so the article hedges rather than asserts. For the "not legal advice" note and what is out of scope, see the disclaimer at the top.

About the author

More than 20 years of electrical and software development — from control engineering at a major electronics manufacturer — plus about 10 years of solo development. Across hardware and software, and across enterprise and individual work, I publish the basics that "become a risk if you don't know them," and I plan to cover practical ways to use AI as well. More at About this blog.