AI scrapers are making the open web harder to maintain

Large-scale scraping activity to collect training data for LLMs and related projects is generating excessive traffic on websites, making the open web difficult to maintain.

Attackers use Residential Proxy networks to send requests from millions of unique IP addresses. They minimize the number of accesses per IP to disguise their traffic as that of ordinary users' browsers, and features such as not loading images or CSS are characteristic, but by the time these are detected, blocking often comes too late.

These proxy networks fall broadly into two types:

• Criminal networks: These hijack ordinary users' devices infected with malware and operate them under the direction of command-and-control (C&C) nodes. Recently, poorly secured media streaming devices have been identified as a major vector.
• Business networks: These claim to be "ethically sourced," obtaining users' network connection permissions through services such as VPNs, and then sell that access for the purpose of bypassing websites' access controls.

These attacks are imposing enormous costs and technical burdens on website operators.