Linux Foundation Says AI Scrapers Overwhelm git.kernel.org with Billions of URLs

The official said that across five geographically distributed nodes, 14 CPU cores are permanently occupied with rendering git commits as HTML for scrapers. The kernel repository linux.git contains about 1.48 million commits, and git.kernel.org hosts about 922 forks of it. Because forks share most objects, they are efficient for humans, but a scraper can generate billions of valid URLs by requesting patches, plain renders, or diffs between arbitrary commits across every fork. The director noted that a single fork of linux.git can produce an estimated 1.2 "metric bajillion" valid URLs.

According to the Linux Foundation, the problem evolved from easily identifiable bots to sophisticated residential IP rotation. Initially, scrapers identified themselves through user-agent strings, allowing administrators to block them with fail2ban. Later, bots impersonated plain browsers, then spread across subnets, and eventually began coming from millions of random residential or mobile IPs that would make four or five requests and vanish. The director likened the behavior to swarms of locusts, describing a cycle in which crawlers hit hard until the system fails, move on, and then return. This pattern is tied to what the director called "proxy SDK monetization," a business in which devices such as TVs can be used as proxies.

The scale of the problem is significant. According to the director, git.kernel.org now receives about 6 million daily requests seeking random commits. Of those, 66% are rejected by the Anubis anti-bot challenge, but 33% successfully solve the math challenge and reach the main site. The director estimated that legitimate requests account for only about 2% of total traffic, with everything else coming from scrapers.

In response, the foundation has begun turning off features to reduce the number of crawlable URLs and to block costly actions. The director said users should expect to lose some functionality when accessing resources anonymously. The foundation still promises to offer all data for download to anyone who asks, but noted that people may have to "jump through more hoops" to obtain it. The director expressed regret over the measures, calling them a necessity under the current conditions.

The Linux Foundation's comments highlight the broader impact of AI training data collection on public infrastructure. By scraping large open-source repositories, companies may be consuming resources needed for human users, prompting maintainers to impose stricter access controls.