Skip to content
Source: Ars Technica

☁️Microsoft Outage Traced to Hidden Gateway Defect, Lasted Nearly 24 Hours

Your Microsoft 365 uptime promises just got harder to trust

TL;DR

A hidden defect in Microsoft's cloud gateway caused one of the company's largest outages, lasting just under 24 hours and freezing over 80% of affected gateway servers. Anyone relying on Exchange Online or Teams should review their failover plans now, because the two-gateway setup didn't save them.

A hidden bug in Microsoft's cloud gateway caused one of the biggest outages in the company's history, and it ran for almost 24 hours. Businesses and public offices that depend on Microsoft 365 and Exchange Online felt it immediately. The company had two independent gateways for resilience, and both still went sideways. If you run anything on Microsoft 365, this one is a wake-up call. The root cause was a change to the gateway that, under certain conditions, exhausted CPU and memory until servers stopped responding. Pre-release testing missed it because the test hardware was weaker than systems already in production. Your staging environment may be lying to you in the same way. The timeline: the outage ran from November 12 at 16:20 UTC to November 13 at 16:10 UTC. About 15 hours earlier, Teams traffic was moved from the alternate gateway onto the primary, which carries Exchange Online access, and capacity was running 14 percent below optimal. Restarts didn't help, and a targeted fix from Microsoft engineers was needed. Microsoft extended email delivery timeouts so delayed messages wouldn't bounce. The full final report is due November 20.

Microsoft Outage Traced to Hidden Gateway Defect, Lasted Nearly 24 Hours — Ars Technica

Key Points

1

The outage ran from November 12 at 16:20 UTC to November 13 at 16:10 UTC, just under 24 hours, tracked under incident IDs EX1550543 and MO1550821.

Keep reading — it's free

Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.

Also get
or

Already a member? Sign in

Microsoft AzureMicrosoft 365Exchange OnlineOutageCloud Gateway

Comments

Subscribe to join the conversation...

Be the first to comment