🛠️Linux Kernel Git Burns 14 CPU Cores Feeding AI Crawlers
TL;DR
Konstantin Ryabitsev reports that git.kernel.org now spends more CPU rendering commit pages for scrapers than on all legitimate access combined. That includes git clones.
Konstantin Ryabitsev reports that git.kernel.org now spends more CPU rendering commit pages for scrapers than on all legitimate access combined. That includes git clones. Simon Willison flags the same exposure for any project serving a large crawlable surface, Datasette included.
Key Points
Across 5 geo-distributed kernel.org nodes, 14 CPU cores at any moment do nothing but render git commits as HTML for crawlers
Scraper load now exceeds all legitimate access combined, including git clones, on the official Linux kernel repository
Willison raises the same concern for Datasette, which exposes a large number of generated crawlable pages by design
Original write-up by Ryabitsev at people.kernel.org, surfaced via Hacker News on 7 September 2026
Why It Matters
Training-data crawlers have become a real line item in infrastructure budgets, and anyone serving generated pages should be measuring scraper CPU before it quietly outgrows real traffic.
Quick Facts
Frequently Asked Questions
Why does this matter?
Training-data crawlers have become a real line item in infrastructure budgets, and anyone serving generated pages should be measuring scraper CPU before it quietly outgrows real traffic.
What happened?
Konstantin Ryabitsev reports that git.kernel.org now spends more CPU rendering commit pages for scrapers than on all legitimate access combined. That includes git clones.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,467 builders reading daily.