📧Microsoft Email Outage Lasts 15 Hours After Lock Contention Bug Spreads Fleetwide
Your rollback plan just got a lot more important
TL;DR
A July software defect in Microsoft 365 network gateway servers triggered a 15-hour email outage that knocked out over 80 percent of servers at its lowest point. Rollback wasn't an option because the defect was months old. The lesson for anyone shipping to large fleets: load tests on weak hardware can hide failures that only appear on modern machines.
Microsoft's email service went down for 15 hours because of a bug that had been sitting in its network gateway servers since July. The update passed load testing and shipped without visible impact. Then the bug surfaced, and it took most of the fleet with it. If you run services on shared infrastructure, this one is your problem too. The testing focused on weaker hardware with fewer cores, but the bug caused lock contention inside an internal Windows process that got worse on machines with more cores and newer hardware. Your staging environment probably looks more like Microsoft's test rigs than its production fleet. Check that before your next rollout. The numbers are rough. Within two hours the defect was hit across Microsoft's fleet, and more than 80 percent of servers were unresponsive at the worst point. Rolling back was impossible because the defect was months old, so engineers built a patch for the patch and applied it group by group. Connection negotiation was the main failure point, with competing processes blocking connections. Mitigation landed at 0720 UTC, and the final customer update went out at 0415 UTC on November 14. A preliminary report is out, and the full post-incident report is due November 20. Watch that document for the details on how the July change got past load testing.

Key Points
The defect was introduced in July in network gateway servers and passed load testing because the tests ran on weaker hardware with fewer cores.
Keep reading — it's free
Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.
Already a member? Sign in
Comments
Be the first to comment