r/technology
•
u/TeamAlphaBOLD
•
5d ago
•
1
Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide
https://cybersecuritynews.com/microsoft-outage-incident-report/
19k
Upvotes
11k
u/lordnacho666 5d ago edited 5d ago •
Ah it was a hidden bug? No wonder.
As long as it isn't one of those visible bugs, I hate those.
254
u/KebabAnnhilator 5d ago
Claude, Fix this code, definitely no mistakes this time
1.7k
u/duckbigtrain 5d ago
> The disruption lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13
Someone was desperate not to hit the 24-hour mark
1.7k
u/albaMP4 5d ago
Are they saying the outage lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13 to avoid some penalty for being down 24 hours?
713
u/j01101111sh 5d ago
So... It wasn't DNS?
950
u/nugscree 5d ago
They forgot to add "do not create bugs or errors, it must work" to their prompt when creating the November patch.
3.2k
u/Character-Handle-697 5d ago •
As a software engineer, here is how to reproduce:
2.4k
u/FaultofDan 5d ago •
So half of the corporate world relies on one company's infrastructure, and that infrastructure is built on dominoes. Sick.
34
u/fresh-dork 5d ago
lol "hidden". azure has been stitched together with hope and fear for 5 years or more. this was inevitable and it will happen again
10
u/Flyerone 5d ago
Hidden bug huh? Cool. I dare you to make us hate you more M$
How long until we're back to an abacus and charcoal on papyrus?
172
u/grantrules 5d ago •
What is an unhidden bug? A feature?
47
u/postinganxiety 5d ago
No way the outage was less than 24 hours. Are they saying that to avoid paying out?
72
u/Organic_Noise8027 5d ago
I just want the word and excel loaded on my laptop without the cloud.
53
u/Phainesthai 5d ago
Partially caused by a dodgy Windows update on some of their servers?
Hmm, they really should be using Linux on their critical infrastructure...
127
u/RileyGainesHorseBaby 5d ago
I can't imagine having always-online DRM for such mission critical stuff. This should be a wakeup call to the ignorant and careless.
161
u/GlassSquirrel9649 5d ago
"hidden windows bug" is corporate speak for "some senior engineer documented this exact failure state in 2018, management marked it as won't-fix in the backlog, and then we laid off the only guy who knew how to patch the gateway cluster." peak enterprise software management
2.8k
u/danivus 5d ago •
So according to this article they missed it because they tested on older hardware than what their live servers use, and the bug only strikes on systems with more cores than their test systems.
Not testing on matching hardware for something so critical, for a company as big as Microsoft, seems... very stupid?
87
u/AndyKJMehta 5d ago
Aren’t all bugs hidden till they’re found?!
68
u/garden-wicket-581 5d ago
TFA really doesn't give you anything useful or meaningful about the bug -- "lock contention" and "resource exhaustion" (with an added scoop of "automatic recovery overpowered the remaining servers")
34
u/Beachhouse15 5d ago
Microsoft 364.35
109
u/sengirminion 5d ago
Dang. Maybe they ought to hire like 60 thousand employees to help dig through their multi-billion dollar lines of code to prevent this from happening again. Just a thought.
43
u/reversethrust 5d ago
lol. Just read the very terse description. I encountered something similar previously - multiple producer, single consumer problem. The queue worked fine on a laptop, but toss that software on a big server with 256 cores. The test went from taking minutes to taking overnight despite the vast amounts of addition hardware 😂. System utilization on the big machine was much less than 1% though in my case. I wonder what it was like on these exchange servers.
33
u/Nings777 5d ago
It was Clippy with a pitchfork
67
u/Imd1rtybutn0twr0ng 5d ago edited 5d ago
And right below this post, I have an ad for them for AI... irony at its best.
Microsoft, forget AI. Focus on creating workable patches and less bloat. Apologies in thinking you want your consumers to like using your product...
86
u/arekitect 5d ago
Now it all makes sense. Microsoft must have installed Teams on the Exchange servers. Everyone knows Teams can suck the life out of even the most powerful computer. Apparently, it finally found a machine it could bring down faster than my laptop. Unfortunately, that machine was Microsoft's entire cloud infrastructure.
36
u/Emotional-Inside3126 5d ago
Bug? Or borged patch ?
7
u/Producer_Earth 5d ago
Someagentbrokeinthesystemandexploitedthedayonebug cough cough
19
u/brattysweat 5d ago
Lucky those planes grounded to get everyone to forget this even happened
54
u/PressureBeautiful515 5d ago edited 5d ago
How many times are we going to read a sentence like "the system had built-in failover for just this eventuality, but unfortunately that just caused the problem to spread and made everything worse" or words to that effect? It seems to be quite a common theme in these postmortems.
Sometimes the problem gets way worse because of how the system is designed to try to reroute and keep working, to be "fault tolerant." Instead it ends up "fault aggravating."
EDIT: diagramofplanewithreddots.jpg
13
u/NoBonus6969 5d ago
I hate it when bugs don't announce themselves at production
35
u/enthusiastic_duck 5d ago
All eyes are now on Microsoft c-suite who decided to lay off senior engineers in hopes to promote AI. The bug was there since ages, but now they have limited expertise to navigate through such issues
6
u/TyrusX 5d ago
All bugs are hidden. But some are known in advance.
6
u/Iron_Wolf123 5d ago
Let me guess, Win 11 24H2 update?
24
u/SnooOpinions8790 5d ago
In 2014 Microsoft abolished its dedicated test practice
In the following 5 years I slowly removed all Windows computers from my house and moved services to other providers. It was always just a matter of time before this happened.
When OpenAI started looking elsewhere for their cloud last year despite their close contractual ties to Microsoft it was obvious that the Microsoft cloud stuff was teetering on the brink.
I will point out that having completely lost the knowledge and skills to test "black box" software they were then in a particularly awful position to start using AI to generate code that none of their engineers have a deep understanding of. But that may not be the immediate cause of this outage - which is the cumulative crap of a decade of declining software engineering skills and bullshit managers rising to the top because there is no dedicated bullshit-detecting practice left in the company.
19
u/dorkmuncan 5d ago
"The disruption lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13."
Suspiciously 10 mins under 24 hours.
15
u/cornbread6767 5d ago
Good thing you just fired 50,000 devs 🙄
45
u/MothChasingFlame 5d ago •
So it was AI and bad processes established by managers with no brains. Got it.
18
u/PurpleCaterpillar451 5d ago •
Bugs are always hanging around windows
5
u/SansSariph 5d ago
I've joked before that just like "everyone" should hold a retail job at least once to learn about it, "everyone" should work a week as an on-call engineer for large scale distributed services.
The weird problems with data, load balancing, and the chain reactions you can cause by recovering, the ability to DDoS yourself with your own clients retrying requests, the entire problem space of dealing with a never ending stream of input that you "keep up with" - it's hard.
Working on services changed my mindset about retrospectives and root causing problems. It's (almost) never just one thing's or one person's "fault" except at a surface level.
25
u/SaveDnet-FRed0 5d ago
TDLR: Microslop's AI generated code base crashed leading to automated backup systems activating that spread the problem to unaffected servers and no one at Microsoft was competent enough to realize what was going on until days after the problem became to big to ignore.
...This also implying that a repeat incident is much more likely to eventually happen then not. Anyone relent on Microsoft should look to switch to an alternative ASAP.
17
u/IncognitoAnonymous2 5d ago
Reports like this are called "damage control"
11
u/Thisappistrash69 5d ago
Linux needs to become a more household name. I think it’s time to move away from windows. We need alternatives
37
u/ZapDapper 5d ago
Hidden just means that they didnt actually look for the case where it happens.
"I closed my eyes and saw nothing"
4
u/FrankieMint 5d ago
Microsoft 364, 363, 362...
4
u/silverwoodchuck47 5d ago
TLDR: Microsoft tests their software in production.
4
u/Complete_Entry 5d ago
It's always a fuckup update and then they roll back the statement.
10
u/Splendid_Carpark 5d ago
A Microsoft product with a bug? Hard to believe.
12
u/castlite 5d ago
The bug is named Copilot
7
u/romulof 5d ago
This is the first time a bug was hidden. They are usually stamped in the facade by developers.
15
u/KraljZ 5d ago
And yet the bug wasn’t found by AI
11
u/unexpectedreboots 5d ago
Man the comments lol. The change was running in production for over 3 months without issue.
A very specific series of events caused it to be exposed. Read past the headline.
12
u/Hightower840 5d ago
Earlier this year MS said that 20 to 30% of it's code was now AI written...
6
u/buzz_light365 5d ago
Where's the link to Microsoft postmortum RCA? This article doesn't have a source. Is this just guesstimates and assumptions from the author??
6
u/Privateer_Lev_Arris 5d ago
A bug in Microsoft? Can't be. /s
8
u/blur410 5d ago
You would have thought that they could have sent an intern to the Best Buy and picked up a new computer. /s
57
u/Major303 5d ago edited 5d ago
First they claim AI is perfect and flawless and people are obsolete, and now they say something about "hidden" bug. Make it make sense. By their logic AI should catch it effortlessly.
I'm not an average anti AI person, but AI tech bros have abandoned common sense.
3
u/HotBBQWin 5d ago
Good thing they got us updating every 5 days
3
u/gormami 5d ago
This reads very much like some outages that happened at Verizon Wireless during my tenure there, and is what I expected, honestly. I'm not as familiar with the internals, but it looks like a control plane storm event. Connections cost too much, and then it tips over where the time to service exceeds the timeout, resulting in a state where no real work is being done. The only way to handle it in the end is to shut it down, then bring it back one chunk at a time to let the initialization work get done and settle back down.
This is perhaps the biggest challenge of scale engineering, as there may be dozens to hundreds of different types of activities with their own timeouts and retry algorithms. Aligning all of them properly is very hard, and changes may be made constantly. Each node needs to protect itself in such a way that some real work gets done for each cycle it spends to have any chance of the system recovering itself. Mostly, that means tracking the time spent, and rejecting new requests if the value exceeds the known timeout. Northbound, applications need to back off, rather than accelerate, when undergoing problems, to keep from overwhelming the system. Even then, constructive interference from multiple processes can make it very hard to get it back under control.
3
u/Tupperbaby 5d ago
It's not hidden anymore.
3
u/NapierNoyes 5d ago
Oh, so they basically DDOS’d themselves.
3
u/martixy 5d ago
Ah, it seems to have been an old-school complex system failure and not part of the AI bugpocalypse.
How refreshing.
3
u/Arkaqiu 5d ago
As I understand, race condition on shared resource. Its interesting how choosing proper test harness and environment is crucial.
Butazzo in one of his book shows how one solution is schedulable on 2 cores but it failed on 3 cores. One
3
u/HowsItBeenBen 5d ago
With all the other stuff thats been goin down lately, I call bullshit.
3
u/LocalObligation5642 5d ago
Took down or offered new opportunities to increase capitalization?
3
u/Adderall_Rant 4d ago
I knew it. It was a weather balloon.
3
u/Nosbus 4d ago
Clippy strikes back...
18
u/OffalSmorgasbord 5d ago
So AI wrote the code and AI tested the code and AI monitoring missed the problem and the humans let go for AI to make the shareholders happy weren't there to fix what AI did...
"We will just sue because management didn't disclose how dependent the products are on AI" - Institutional Investors
5
u/sadmistersalmon 5d ago
one hidden bug was the root cause. But as you read the rca, you see multiple instances of other “unrelated issues” with corresponding recovery and compensating mechanisms fighting to restore operations. it’s as if you have a heavily leaking bucket that you are trying to keep full by pumping water non stop.
5
u/jcauch 5d ago
I've come to hate this company so much. They've been around for 50 years yet Powerpoint can become unstable when trying to copy paste. They'd never gain market share today with the poor product quality they deliver. Make the simple stuff work. Stop with the slop.
5
u/InAppropriate-meal 5d ago
In other words they pushed some slop-coding and it screwed everything up on an impressively large scale 😂
14
u/FlamingoOk3026 5d ago
Never use windows for mission critical applications.
15
u/Fickle_Competition33 5d ago
The fact the outage was global, not regional, tells a lot how amateur the whole ecosystem is. You can badmouth AWS as much as you want, but their outages and design are always region-bound.
7
u/msc1 5d ago
"Country Blocked
Access to this page is forbidden."
If you can't deal with script kiddies around the world, maybe don't host cyber security news.
10
u/julioqc 5d ago
so they say... god forbid the tech they invested billions into blindly has not delivered promises
4
u/rigsta 5d ago
Nah I'm sure it was just a new router
4
u/wretch5150 5d ago
Yeah, outsourcing everything including development on windows, then vibe coding m365 updates will do that.
4
u/strcrssd 5d ago
So they developed an internal only patch, deployed it, and it failed. Not mainlined or beta dogfooded, but code that they see as crucial for their operations that they don't ship to customers.
Another reason to stay away from MS. They don't even run their own stock software.
8
u/Sans-valeur 5d ago edited 5d ago
Man when I started investing I thought, it’s Microsoft. How much of the world runs on their products? As safe a bet as any in terms of single stock investments.
I mean, it hasn’t been a bad investment, but yesh, this decade….
The fuck are they doing?
I bought my first Mac ever last year, and I’d never even really considered it before.
This is one of the inherent problems with late stage capitalism.
The goal isn’t to create something good, and then keep it good.
It’s relentless, constant growth by any means necessary, gambling with the entire fucking company if need be.
Constantly looking for the line, how much you can reduce quality while maximizing profit before you drive away your customer base.
5
u/junktech 5d ago
Hey, Microsoft, do you feel like disclosing the backlog? Like the real one with silenced alerts. Not the official one.