Microsoft pins Exchange Online outage on July code defect
TL;DR
- Microsoft's preliminary post-incident report blames a July 2026 update to network-gateway servers for last week's multi-day Exchange Online outage.
- The lock-contention bug surfaced only on high-core-count hardware; Microsoft says its load testing focused on weaker machines with fewer cores.
- A standard rollback was impossible after three months in production; at peak, more than 80% of networking gateway servers were unresponsive.
Microsoft's preliminary post-incident report traces last week's multi-day Exchange Online outage to a latent lock-contention bug that had been sitting inside its network gateway servers since a July 2026 update, according to The Register. The defect only surfaced on higher-core-count hardware, where lock contention in a Windows process exhausted CPU and memory.
Two conditions sprang the trap on November 12. Teams traffic failed over onto the primary Exchange Online gateway at 01:07 UTC, piling on unusual load, while approximately 14% of gateway server capacity was temporarily offline. Affected servers stopped responding and routed traffic to other servers hitting the same wall, cascading through the fleet until over 80% of networking gateway servers were unresponsive.
"Our load testing tended to focus on weaker hardware with fewer cores," Microsoft wrote in the report, acknowledging that the fault grew worse on newer, beefier machines rather than better.
A conventional rollback was off the table because the defective code had been in production for three months. Engineers instead wrote corrective code to restore the previously functional paths, deploying what the report describes as a "patch that reverted the defective change" at 07:20 UTC on November 13; services were declared healthy at 16:10 UTC. Teams, OneDrive, SharePoint and Office apps rode out the incident on alternate infrastructure, which AI Weekly tracked as the outage unfolded.
Microsoft's normally automated recovery was not designed to restore service from this broad unresponsive state, the report notes.
Originally reported by theregister.com
Read the original article →Original headline: Microsoft blames November Exchange Online outage on a software defect introduced in a July update