Blog

Web data collection

Articles tagged "Web data collection".

RightsMay 28, 2026

Defending Your Site from AI-Training Crawlers — robots.txt, noai & CDN Blocks (US/EU)

Only a CDN-level block actually stops an AI-training crawler; robots.txt, X-Robots-Tag, and noai signal intent and leave a record. A four-layer defense stack, host-by-host setup, and where US fair-use and EU DSM litigation now stands.

Web data collection Copyright Infrastructure

RightsMay 25, 2026

The Legal Boundaries of Web Scraping for US/EU Builders — CFAA, GDPR, DSM & AI Training

A US/EU-focused guide: when you scrape — or let an AI fetch pages — sort your case into four legal lenses (CFAA, copyright/fair use, GDPR, contract) before you use the data. Covers hiQ, Meta v. Bright Data, the Ross/Bartz/Kadrey AI-training split, and Clearview.