Microsoft traces its mail outage to a July fix
A Windows change Microsoft began deploying in July sat dormant for more than three months before taking down the gateway in front of Exchange Online, its preliminary report says. More than 80 per cent of the gateway's servers stopped responding.
J.smith, CC BY-SA 2.5, via Wikimedia CommonsA Windows fix that Microsoft began deploying in July caused the Exchange Online outage of 12 and 13 November, the company says in a preliminary post-incident report dated 15 November.
A surge of Teams traffic and a block of servers already out of service pushed the gateway in front of Exchange past breaking point, the report says. More than 80 per cent of the gateway's servers stopped responding at the worst point, and mail access was disrupted for most users for 23 hours and 50 minutes.
The seven-page report, a copy of which was published by the Microsoft 365 site 365Fanatic, describes "a latent defect" that made Windows servers "overwhelmed and unresponsive under certain load conditions" through "lock contention inside an internal Windows process". The fix was "prepared specifically for Microsoft 365 networking gateway servers", and Microsoft says "neither the underlying defect nor the fix applies to broader Windows deployments". The report gives the outage as 16:20 UTC on Thursday to 16:10 UTC on Friday, 23 hours and 50 minutes by our arithmetic.
Why did a July change fail in November?
Because the load changed. The report says the change "first started deployment in July of this year" and that "For more than three months, the change was deployed worldwide without impact", between 104 and 134 days by our count depending on the July start. Load testing missed it: "The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores."
Two things then lined up. About 15 hours before the outage, at 01:07 UTC, Teams failed over from Microsoft's alternative gateway onto the "primary" one that also carries Exchange Online, which the report calls "part of normal operations". And "nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day". The servers began failing "during the peak window of traffic in the United States". When failing servers handed their traffic to healthy ones, those failed too: "In less than two hours, the previously unknown defect was encountered across our fleet", and "At its lowest, more than 80% of servers were unresponsive."
| Time (UTC) | What Microsoft's preliminary report says happened | Hours after the 16:20 alert |
|---|---|---|
| 01:07 Thu 12 Nov | Automated failover moves Teams traffic onto the gateway servers shared with Exchange Online | −15.2 |
| 16:03 | Anomaly monitor sees connectivity problems rising | −0.3 |
| 16:20 | High-priority alert; incident EX1550543 posted | 0 |
| 19:00 | All deployments and configuration changes paused | 2.7 |
| 00:30 Fri 13 Nov | Connection negotiation identified as the main cause of access failures | 8.2 |
| 07:20 | Update reverting the defective change applied to part of the capacity | 15.0 |
| 14:17 | Global webmail availability back to expected levels | 22.0 |
| 16:10 | No active customer impact; extended monitoring begins | 23.8 |
The gateway was built to contain this kind of failure, and that did not work either. Microsoft says the gateway is made of "thousands of 'capacity units'" grouped into regions, but "due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure", which is how the fault spread worldwide. The timeline shows the first night's fixes failing: at 22:00 UTC on Thursday Microsoft's telemetry "confirmed that the actions we'd taken to restore service had not been effective", and at 23:55 it found that servers returned to service "initially remained healthy but then become unresponsive again upon receiving traffic". The report does not describe the separate issue that weakened the bulkheads.
Teams, OneDrive and SharePoint moved off the faulty gateway, which is why they recovered first, but "not all services, like Exchange Online, can fail out of this network gateway for scale reasons". Restarted servers were "immediately overwhelmed by waiting traffic", and because the defect was months old, "our normal rollback, flighting, and feature flags were unable to quickly revert the change". Engineers took eight hours and ten minutes from the first alert to identify connection negotiation as the problem, and 15 hours to begin applying an update that put the code back to its earlier state, by our arithmetic from the report's timeline.
Which start time is right?
The report says 16:20 UTC, but Microsoft's own incident record still disagrees. The closed record for EX1550543, as copied by the University of Pennsylvania on Tuesday, carries a Start Time field of 10:05 UTC, the time Komposite reported on Saturday as a six-hour revision. Its final status text and the report both say 16:20. The record's status changed to "postIncidentReviewPublished" with a note at 04:47 UTC on Monday that "A post-incident report has been published".
| Where the start time appears | Start (UTC, Thu 12 Nov) | Outage to 16:10 Fri | Uptime for a user out throughout, 30-day month |
|---|---|---|---|
| Incident record as first logged (our 13 November reading) | 16:17 | 23h53m | 96.68% |
| Start Time field of the closed record, still shown on 17 November | 10:05 | 30h05m | 95.82% |
| Final status text and preliminary report | 16:20 | 23h50m | 96.69% |
For customers' service credits the difference does not change the tier, by our arithmetic, though it changes the minutes counted, which Microsoft measures per user across a tenant. As we reported on Friday, customers have to claim those credits from Microsoft; they are not paid automatically.
The report lists five repair items as completed, including guardrails "for traffic distribution to avoid cascading impact" and more monitoring, and says a "comprehensive list of repair items will be delivered as part of the final PIR" on 20 November, the date The Register also gave. 365Fanatic's summary says the report runs to seven pages. The final report is the next fixed date, five business days after Friday's end time.