Abusive Crawlers Overwhelm Linux Kernel Git Mirrors

Konstantin Ryabitsev has raised concerns about the escalating "background radiation" caused by abusive web crawlers targeting the official Linux kernel Git repository (git.kernel.org). In his recent remarks, Ryabitsev noted that the CPU resources devoted to rendering commit pages for scrapers now exceed those used for all legitimate traffic, including standard git clone operations.

He quantified the issue by stating that, across five geographically distributed nodes, 14 CPU cores are dedicated solely to generating HTML views of commits for these automated agents. This level of resource consumption is unprecedented and threatens the efficiency of the infrastructure that supports kernel development.

The problem resonates with other projects that serve large numbers of crawlable pages. Ryabitsev mentioned his own experience with Datasette, a tool that similarly faces heavy crawler traffic, underscoring that the challenge is not isolated to the Linux kernel ecosystem.

The discussion was highlighted on Hacker News, where the community is tracking the impact of such abusive crawling behavior on open-source infrastructure. The source of the commentary is Simon Willison’s blog post dated September 7, 2026.

Addressing this issue may require new rate‑limiting strategies or better bot detection mechanisms to preserve CPU capacity for genuine developers and users of the Linux kernel source code.