Linux Kernel Repository Overloaded by AI Crawlers' HTML Rendering Load

The operators of git.kernel.org have disclosed severe server load issues caused by AI crawlers. Currently, 14 CPU cores across 5 geographically distributed nodes are being used solely to render git commits as HTML, consuming more CPU cycles than all other legitimate access methods, such as git clones.

Inefficient Data Collection by Crawlers

The Linux development process is open source and highly valuable as a pure data source free of AI-generated content for LLM training. However, crawlers are adopting an inefficient approach, rendering and parsing 1.48 million commits and 922 forks one by one as HTML pages instead of using the efficient git clone. This exploits cgit's flexibility to generate billions of valid URLs.

Threats from Block Evasion and Proxy SDKs

Initially, crawlers were blocked based on User-Agent or IP addresses, but bots have evolved to masquerade as regular browsers and bypass large ASNs like Google Cloud. Recently, they have been using techniques that make 4-5 requests from millions of residential or mobile IPs before disappearing, which is linked to the 'proxy SDK monetization' business installed on devices like smart TVs. These distributed attacks render firewall rules meaningless and persist by attacking again once the system recovers.