r/technology • • 5d ago • Bravo 1

Software Microsoft Reveals How a Hidden Windows Bug Took Down Microsoft 365 Services Worldwide

https://cybersecuritynews.com/microsoft-outage-incident-report/
19k Upvotes

1.5k comments sorted by

11k

u/lordnacho666 5d ago edited 5d ago • Bravo Mindblown Regret

Ah it was a hidden bug? No wonder.

As long as it isn't one of those visible bugs, I hate those.

1.5k

u/TeamAlphaBOLD 5d ago • Bravo

Next they’ll ask us to pay extra for the visible ones.

617

u/BasvanS 5d ago • Bravo

I think that’s the business model already

→ More replies (13)

52

u/Consumption2Wombly 5d ago

Rude way to refer to Windows 11

→ More replies (19)

1.0k

u/Hrekires 5d ago

Not gonna lie, one of the most frustrating days at work was when my team got absolutely reamed out by upper management over an outage caused by a bug that we were one of the first customers in the world to experience.

It was so new that the vendor hadn't even released an advisory for it much less a patch to fix it. There was literally nothing we could have done to prevent it.

170

u/shea241 5d ago

When I was trying to find my place in 3D rendering 22 years ago, I had just started working with a small team in Atlanta on a film project that was far too ambitious. The very first thing I made, a tool to copy 3D data from one character and paste it onto another, would crash if used in a certain edge case. It was a clear bug in Maya (one of many), but the owner of the studio fired me the next day for 'messing up the pipeline'. There was no pipeline.

The owner was absolutely insufferable, came to find he had been burning through employees almost as fast as he'd been burning through investors' cash. It was an important experience finding out people like that exist.

→ More replies (2)

297

u/Leon_Flux 5d ago

When pistons were introduced to minecraft one of the staff in a server I was working with decided to put a pressure plate on it.

corrupted the world.

We got so much shit for something that we had no way of knowing it was a problem. Two of the mods we had in call were crying. It was bad.

→ More replies (11)
→ More replies (15)

518

u/xtothel 5d ago • Bravo Mindblown

The visible ones are features.

186

u/Harm101 5d ago

// Deal with this later. John H intern 11/16/2006

→ More replies (18)

65

u/piss_artist 5d ago

They call it Copilot

→ More replies (3)
→ More replies (8)

119

u/Wit-wat-4 5d ago

I know you’re joking, but there absolutely are many existing bugs that we just live with either through bandaids (duct tape if you’re a plane!), and they do sometimes fail too.

As a quality investigator it’s both good and bad when it’s a hidden bug: worse investigation, but at least you’re not going “god dammit I told them” during the outfall lol.

→ More replies (8)

126

u/junktech 5d ago

Hidden translate roughly to : Some senior documented it but due to internal chaos it's been in forgotten in the backlog. Eventually management acknowledged it so it doesn't show alerts.

→ More replies (1)

342

u/TheComplimentarian 5d ago edited 5d ago

The bug they’re describing I’ve seen in windows since 2008, and it wasn’t new then.

Its natural response to a node going down is trying to shunt all that traffic (regardless of whether it’s needed or not) to every other node, along with a lot of “NODE DOWN!” traffic, so minor health issues can cause stuff in the AD to essentially DDoS itself with panic traffic.

Gracefully adding and removing nodes (or scaling at all, frankly) is not something windows is great at.

55

u/Left_Contribution833 5d ago

The bug isn't the shunting. That's normal experience. The bug was a specific condition that led to resource locks. And that was applied all over the servers, so the moment traffic got shunted, the new server ran into the lock faster.

→ More replies (6)

88

u/uptwolait 5d ago

That sounds oddly similar to anaphylactic shock in the immune response.

→ More replies (4)
→ More replies (12)

30

u/spotcatspot 5d ago

New guy didn’t check show hidden bugs in the view pulldown.

→ More replies (1)

11

u/MuscaMurum 5d ago

The MSFT Windows bug database had thousands of bugs resolved as "Won't Fix" last I knew. Every product has those.

22

u/JeebusChristBalls 5d ago

Visible bug would be worse because it meant they knew about it. A known bug doesn't mean that it is going to be fixed anytime soon. It is just known. You know, the opposite of unknown, ie. Hidden.

13

u/Ziatch 5d ago

There are plenty of bugs that aren’t hidden though so I don’t get the smugness

→ More replies (35)

254

u/KebabAnnhilator 5d ago

Claude, Fix this code, definitely no mistakes this time

8

u/TeamAlphaBOLD 5d ago

This cracked me up.

56

u/IAmBadAtInternet 5d ago

Copilot malding rn lmao

→ More replies (4)

4

u/likamuka 5d ago

Michael, you are a shining beacon of intellect! Here is the code I have just fixed. Ms Office 98 is running as it should now.

→ More replies (1)
→ More replies (1)

1.7k

u/duckbigtrain 5d ago

> The disruption lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13

Someone was desperate not to hit the 24-hour mark

1.2k

u/Standard_Wish_9338 5d ago • Bravo Popcorn

Otherwise they would have to rename it to Microsoft 364

152

u/sb969 5d ago

They get a free crash day in leap years.

→ More replies (7)

179

u/nolabmp 5d ago

Their SLA likely has that exact requirement.

100

u/UnexpectedAnanas 5d ago

Good thing the SLA didn't say they can't tell lies!

→ More replies (3)

12

u/PlannedObsolescence_ 5d ago

Late reply but no, the 99% SLA for Exchange Online got breached at 7.2 hrs, and the next threshold is 95% which is 36 hrs.

→ More replies (1)

1.7k

u/albaMP4 5d ago

Are they saying the outage lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13 to avoid some penalty for being down 24 hours?

1.1k

u/TemptedTemplar 5d ago • Bravo

Credit payments. They offer claim able discount credits to anyone who pays for the subscription when it experiences unannounced downtime.

There's a small discount for hour+ outages and a much biggrler one for 24+ hours.

Enterprise admins would have had a huge payday.

92

u/FUBARded 5d ago

It won't be as simple as them being able to arbitrarily set outage timestamps that allow them to significantly reduce the penalties they're on the hook for.

If these loopholes are painfully obvious to us, they certainly would've been to the many lawyers on both sides that would've reviewed some of these 7, 8, or even 9 figure contracts.

To be clear, I don't doubt they pulled some bullshit to minimise their losses; I'm just saying it's not as simple as fudging a timestamp in a press release. For example, there's certainly a very detailed set of criteria they need to meet to reset the outage timer, and they probably had their engineers working to prioritise doing just enough to meet those criteria even if it means a full recovery of service took longer.

Acceptance criteria for fees at risk, penalty provisions, and tiered discounts will always be gamed – that's the nature of these commercial structures. Buyers always try to get them as airtight as they can of course, but that's not always possible, especially with a supplier in a duopoly market (Google is the only real competitor to M365).

Microsoft probably has more favourable contract terms than most other suppliers given the lack of choice and massive cost associated with switching to e.g., Google Workspace.

→ More replies (4)
→ More replies (8)

423

u/eronth 5d ago

Yeah that sounds outright false. I recall seeing it still down like 26 to 28 hours after initial downtimes.

347

u/WeTheSalty 5d ago

They'll probably split hairs and say something like "well it came up for an hour before it crashed again, so it wasn't a 26 hour downtime, it was two 13 hour downtimes"

→ More replies (14)
→ More replies (7)

142

u/theZinger90 5d ago

Probably some worthless KPI, either internal or imposed by an auditing/consulting group being inflated as more important than it actually is.

121

u/enablingark 5d ago

I was thinking more like SLAs, their contracts probably have penalties for 24 hour long outages or something.

→ More replies (3)

73

u/Shot-Arugula8264 5d ago edited 5d ago • Bravo

They would’ve legally had to rename it M364 otherwise.

Edit: typo

35

u/ForensicPathology 5d ago

rename it M354

It was down for 11 days??

→ More replies (1)
→ More replies (4)
→ More replies (6)

713

u/j01101111sh 5d ago

So... It wasn't DNS?

169

u/Goingone 5d ago

Technically lock contention for DNS lookups was not ruled out by the article.

25

u/FlamingoOk3026 5d ago

For a change

274

u/No-Designer-6533 5d ago

Definitely points more towards it being an AI related bug. Would not be very surprised at all if this whole problem is due to them firing all their experienced developers and just thinking AI will fix it 

→ More replies (93)
→ More replies (8)

950

u/nugscree 5d ago

They forgot to add "do not create bugs or errors, it must work" to their prompt when creating the November patch.

98

u/ZombieFleshEaters 5d ago

Make no mistakes

147

u/IJustWantPleasurePlz 5d ago • Bravo

Don't include any visible bugs.

Got it, hidden bugs it is 👍

→ More replies (3)

3.2k

u/Character-Handle-697 5d ago • Bravo

As a software engineer, here is how to reproduce:

  1. Fire a lot of people
  2. Use AI for make a shit ton of code.
  3. Review millions of lines of code ? No, ask AI to review it.
  4. Keep going for months
  5. Find an issue
  6. Ask AI to find it
  7. AI doesnt find it or find something that it’s not the real bug
  8. Ask your remaining devs to find the bug in the millions of lines
  9. Find a stupid reason to protect the AI reason

1.0k

u/luckless_lord 5d ago

10. Ask AI to write the article explaining what went wrong.

201

u/viral3075 5d ago

also give nonsense AI updates while customers scramble with no comms

→ More replies (1)

69

u/EskilPotet 5d ago

Post it on reddit and let AI upvote it

→ More replies (1)
→ More replies (4)

331

u/UseWhatever 5d ago

100% this. I’ve tried explaining to project managers that while AI is fast, everything gets bogged down in the review process. Which leads to the same or longer total development time.

There are expectations when reviewing human-written code that can’t safely be applied to AI code no matter how good the model is.

86

u/Lucky-Earther 5d ago

100% this. I’ve tried explaining to project managers that while AI is fast, everything gets bogged down in the review process. Which leads to the same or longer total development time.

I've been in the dev world for a long time now, and the writing code part was always, at most, 50% of my job.

→ More replies (5)

100

u/purpleburgundy 5d ago

Hear me out: what if we threw TWO AIs at the pull request and automated-approval process?

Edit: I'm only half-joking, work these days feels increasingly like the children's story There Was An Old Lady Who Swallowed A Fly

→ More replies (10)

14

u/BellacosePlayer 5d ago

I've talked to so many people on reddit who claim to be SWEs and don't do any review anymore in the age of AI.

I can't tell if they're larping or they're working for some shithole startup with 0 users who doesn't care about silly things like outages or data corruption

13

u/decadent-dragon 5d ago

Because somehow we very quickly went from: wow claude is a time saver to here’s a PR with 15 files changed all 100% written with AI and 4 readme’s that are slop

AI is crazy good at some things. Finding bugs, race conditions, writing one off scripts, or isolated functions. But it’s kind of ass at putting together a large project

As a tool to speed up development it’s kind of insane. As a 100% replacement for a dev it sucks

→ More replies (2)
→ More replies (7)

48

u/TerenceMulvaney 5d ago • Regret

Rinse and repeat...

29

u/NerdTalkDan 5d ago

Hey Gemini, please tell me how to rinse and repeat

→ More replies (2)

24

u/MysteryPerker 5d ago

The software code is going to be so AI fucked in 10 years that people won't be able to fix it.

15

u/BassmanBiff 5d ago

To paraphrase Cory Doctorow: sloppy AI code is the asbestos that we're stuffing into the walls of our digital infrastructure, and it's going to be very expensive to remove in order to make things safe later on.

→ More replies (3)
→ More replies (25)

2.4k

u/FaultofDan 5d ago • Popcorn Regret

So half of the corporate world relies on one company's infrastructure, and that infrastructure is built on dominoes. Sick.

910

u/Zarcona_Martana 5d ago

Most tech infrastructure is held together with Elmer’s glue and popsicle sticks… and that’s only half an exaggeration.

130

u/BigMax 5d ago

Right. It's all built with great intentions, and often starts pretty decent.

But it's like building a house that's a pretty good house. Then you add another bedroom in a converted attic. Then another bedroom in the basement. Then you tack on an addition with a bedroom, kitchenette and bathroom. Then you tack on a garage with a spare room above it. Then you tack on another addition with an office to run your little consulting business. Then you add a separate entrance to that for customers, with a little parking lot on the side for that entrance. Then you add another floor above the main portion of the house.

And on and on and on.

And each little addition on it's own more or less makes sense. "Why start over, when all we really need is another bathroom? Now all we need is more parking, so let's just add that on?"

But then 25 years later, your single family home is a janky mess of strange plumbing systems and weirdly connected pieces and inconsistent building standards and design decor, that rambles in a crazy way that doesn't make much sense as a whole, with half of the things being rush jobs and it's incredibly complicated and strange to maintain.

→ More replies (7)

124

u/me_not_at_work 5d ago

Look at this rich guy that got enough budget from C-level bean counters to afford the fancy Elmer's glue.

→ More replies (15)

117

u/Sick_Wave_ 5d ago

I like to compare software to an ever-growing skyscraper. It isn't a sleek, perfectly architected, marvel of engineering. It's more like thousands of trailer houses, each completely different, all welded together just enough that it keeps working. 

→ More replies (6)

48

u/BINGODINGODONG 5d ago

Most of core finance systems are barely held together by excel sheets made by some dude

→ More replies (5)

20

u/Hinermad 5d ago

When I was young I wanted to work in telecom. Having worldwide communications infrastructure with seemingly incredible uptime sounded very sexy.

Then I went to work for a small telecom company and learned the worldwide network is carried on the backs of millions of support engineers with pagers. It was like finding out a Cray X-MP ran on coal.

→ More replies (6)
→ More replies (20)

82

u/BigBoyYuyuh 5d ago

Basically 3 companies control the internet. Amazon goes down you lose half the internet, Cloudflare goes down you lose the other half, Microsoft goes down you lose email etc…

11

u/zzkj 5d ago

You'd be surprised how much depends on Akamai too.

→ More replies (2)
→ More replies (7)

100

u/PTSDaway 5d ago

You'd be terrified if you saw how much technology relies on maintenance by honbyists. All digital infrastructure is a pyramid of cards, upside down.

In 2024 an open-source compression software (Think what winrar does), was infiltrated by a "helper" who did improvements and maintenance. Then one update interacted with internet/router that would allow this "helper" to enter probably every system with this update, effectively every database on the planet. It's not the worst security issue ever, because only individuals could use it. But espionage wise and how it went under the radar - it is perhaps the most dangerous intentional cyber attack attempt ever made.

It was discovered early in the unofficial release by sheer luck when another hobbyist could not make his own system run as quick like it used to - then saw this software interacting with the internet connection, adding a few milliseconds more to the processing time, that was it.

73

u/UnexpectedAnanas 5d ago

You'd be terrified if you saw how much technology relies on maintenance by honbyists. All digital infrastructure is a pyramid of cards, upside down.

Fine, I'll be the one to link the relevant XKCD

https://xkcd.com/2347/

→ More replies (2)

19

u/Farther_father 5d ago

Ironically, that “hobbyist” was a senior Microsoft dev IIRC

→ More replies (9)
→ More replies (4)

65

u/Wit-wat-4 5d ago

If Excel went away the world would crumble.

Switching from outlook to gmail would be annoying but doable. But excel to Sheets? Nope that’s a terror I don’t wish upon the world

28

u/hotdoug1 5d ago

Hell, my company wants us to switch to Macs and Excel is missing some key features that aren't on PC. I'm holding onto my PC telling them I literally can't do parts of my job without it.

→ More replies (17)
→ More replies (20)

51

u/ikonoclasm 5d ago

Not just the corporate world. Pretty much everything in the modern world relies on hundreds of little pieces of open source code maintained for free by a single or handful of volunteers who has no idea how important their code is to vital infrastructure.

51

u/casillero 5d ago

Dude. its essentially a utility. And that's how you should view it. If something happens at hydro in toronto or con eddison in Manhattan, your gonna have a power outage.

If something happens at Gmail at amazon or microsoft... you will have.. an.. outage.

Its the same 'grid concept' except were all connected on this grid globally.

→ More replies (6)

17

u/Akegata 5d ago

That's just part of the story.
A huge number of companies' infrastructure relies on several companies whose infrastructure is built on different sets of dominoes (see the Cloudflare outage last year) that all need to stay up for everything to work.

→ More replies (2)

48

u/Merusk 5d ago

Wait until you figure out most adults have zero idea what they're doing and everyone's really just muddling along.

14

u/HelpfulSeaMammal 5d ago

Nobody knows what they're doing until after the thing goes awry. Then suddenly you're surrounded by experts in niche things like airport communication infrastructure as seen with the whole FAA thing last week lol

→ More replies (1)

25

u/petrichorax 5d ago

the ones who do know what they're doing are screaming until they're blue in the face on how to fix things, but they're getting ignored by the people in charge

→ More replies (1)
→ More replies (1)

42

u/Desperate-Dig2806 5d ago

Everything is more or less built on dominoes.

→ More replies (4)

14

u/NimusNix 5d ago

First time?

10

u/AnsibleAnswers 5d ago

Furthermore, that company uses it's own in house operating system with a notoriously shit networking stack for infrastructure instead of Linux. Every problem they encounter is only their problem.

→ More replies (1)

55

u/calladc 5d ago

no this article is blowing it out

the outage report explicitly states why. this article is misleading.

the explicitly state that the m365 gateway servers run a custom windows operating system and the change that impacted last week has been in since July.

you couldn't run this distro as it's tied directly into the backend management of m365. the fault was specific to that server doing that role. it's custom internal software

→ More replies (14)

8

u/GODDAMNFOOL 5d ago

Aren't you just glad that we're offloading essential government systems like ATC onto corpo-owned cloud services? Surely that won't cause any issues.

12

u/ChinookNL 5d ago

I worked for a bigish european company. we were running the whole business on a AS400

→ More replies (11)

31

u/Cygnus94 5d ago

A lot of people are pointing out the glaring flaw of the hardware discrepancies between test and live being an issue, but why the fuck did automatic recovery keep spooling up new instances of bad code? 

One server would fall over, ok, spool up a new one, hmm it happened again, let's try again. At no point did it question why server after sever was going down. At a certain point it should be detecting a problem and preventing any further rollout of the code, but it just kept going until there was manual intervention.

57

u/calladc 5d ago

Microsoft's actual outage report covers it better than this article. this article is basically just rage baiting.

they knew it was spitting out bad configuration. they didn't know why.

it turned out to be a change from July that only became an issue when a specific network configuration was unavailable due to an unrelated outage.

the systems kept using more compute. when higher spec compute was provisioned the issue got worse.

the network configuration issue had been parked as non critical and the whole environment was under configuration freeze

→ More replies (5)
→ More replies (1)

11

u/KebabAnnhilator 5d ago

Technically half the stock market was at the whims of Jensons genitals but hey, that’s the way the cookie crumbles

→ More replies (41)

34

u/fresh-dork 5d ago

lol "hidden". azure has been stitched together with hope and fear for 5 years or more. this was inevitable and it will happen again

→ More replies (2)

10

u/Flyerone 5d ago

Hidden bug huh? Cool. I dare you to make us hate you more M$

How long until we're back to an abacus and charcoal on papyrus?

6

u/TeamAlphaBOLD 5d ago

100% uptime guaranteed in those.

→ More replies (2)

4

u/JoeDawson8 5d ago • Heartwarming

My baby loves his abacus that I had as a child. We are prepared

→ More replies (1)

172

u/grantrules 5d ago • Bravo

What is an unhidden bug? A feature?

38

u/Lord_Abort 5d ago

One that nobody bothered to fix yet because time line, money, or manpower. 

→ More replies (1)

8

u/AcademicBowl5596 5d ago

You know how when you're using Microsoft products like Windows and Outlook and they all work like a buggy piece of shit that can't remember what timezone you're in even though you've set it manually and even gave it permission to track your location?

Those are unhidden bugs.

→ More replies (6)

47

u/postinganxiety 5d ago

No way the outage was less than 24 hours. Are they saying that to avoid paying out?

30

u/Merlins_Secret_Sauce 5d ago

Yes. The cascade started well before that. We had reports of Outlook AND Sharepoint failures around 2pm. They’re playing “entree is the real meal” with their tickets

→ More replies (1)

72

u/Organic_Noise8027 5d ago

I just want the word and excel loaded on my laptop without the cloud.

21

u/No-Phrase-97 5d ago

Every cloud migration my employer has done has resulted in a diminished service. I can't believe we're still falling for this sh*t 

→ More replies (1)

4

u/arakkal_abu7 5d ago

I just want my chat application to not be sluggish and take me back to dial up era internet.

→ More replies (3)

53

u/Phainesthai 5d ago

Partially caused by a dodgy Windows update on some of their servers?

Hmm, they really should be using Linux on their critical infrastructure...

→ More replies (5)

127

u/RileyGainesHorseBaby 5d ago

I can't imagine having always-online DRM for such mission critical stuff. This should be a wakeup call to the ignorant and careless.

161

u/GlassSquirrel9649 5d ago

"hidden windows bug" is corporate speak for "some senior engineer documented this exact failure state in 2018, management marked it as won't-fix in the backlog, and then we laid off the only guy who knew how to patch the gateway cluster." peak enterprise software management

25

u/TapedeckNinja 5d ago

No, it was a bug introduced a few months ago, in July.

→ More replies (2)
→ More replies (2)

2.8k

u/danivus 5d ago • Popcorn Bravo Mindblown

So according to this article they missed it because they tested on older hardware than what their live servers use, and the bug only strikes on systems with more cores than their test systems.

Not testing on matching hardware for something so critical, for a company as big as Microsoft, seems... very stupid?

54

u/rwv 5d ago • Bravo

The GOAL is for the test environment to match production “as close as possible”.  The realm of possibility includes cost and scale.  

I’m not going to  apologize for Microsoft, but performing testing on older hardware during a use case where historically performing the same type of testing on older hardware has been sufficient seems reasonable.   As a lesson learned they need to ensure that the test environment has systems with a number of cores that match production in the future.  

10

u/Percinho 5d ago

And also, the chances of getting budgetary sign off on a prod-spec test system would be negligable until it was proven that it was an issue. Now there is at least a chance of getting it.

→ More replies (7)

244

u/kyubifire 5d ago

You're not wrong but this is more common than you think, especially now that hardware prices have skyrocketed, it is difficult to maintain parity between a dev environment's hardware and a production environment's.

77

u/BagelsOrDeath 5d ago

And beyond the physical infrastructure costs, running a parallel prod environment can prove very costly both in the effort to ensure parity and the spoofing or duplication of external dependencies and services.

→ More replies (3)
→ More replies (5)

1.8k

u/RustOnTheEdge 5d ago • Bravo

I hate to defend Microsoft but just claiming this is ridiculous. Microsoft’s infrastructure is huge, almost unimaginable huge. There are so many different hardware types at play, it why you abstract it away.

1.0k

u/Historical_Owl_1635 5d ago • Bravo

Anyone who works in tech knows not to take Reddit seriously talking about tech.

→ More replies (124)
→ More replies (114)

30

u/Unusual_Staff_3119 5d ago

Test systems are always smaller that the production system. 

→ More replies (7)

68

u/ConsiderationDry9084 5d ago • Heartwarming Bravo

When ever you feel imposter syndrome, just think back to all the times Microsoft has screwed like this. It does wonders for your mental health.

→ More replies (2)

24

u/Local_Wave4618 5d ago edited 5d ago

Not really.

The point of programming languages is to abstract away the hardware consideration. The programmer shouldn't need to worry about the hardware in most cases.

ETA: Yeah, this was an OS level bug. The Outlook devs aren't to blame for this one.

→ More replies (1)
→ More replies (60)

87

u/AndyKJMehta 5d ago

Aren’t all bugs hidden till they’re found?!

29

u/windsockglue 5d ago

I mean, yes. But there's plenty of bugs that are found and are not fixed before the products are released for a huge variety of reasons. 

16

u/v0id0007 5d ago

Lightning bugs usually light up to show you where they’re at

→ More replies (2)
→ More replies (4)

68

u/garden-wicket-581 5d ago

TFA really doesn't give you anything useful or meaningful about the bug -- "lock contention" and "resource exhaustion" (with an added scoop of "automatic recovery overpowered the remaining servers")

10

u/Adventurous-Mind6940 5d ago

It does say the full report is due Nov 20th.

→ More replies (1)

34

u/Beachhouse15 5d ago

Microsoft 364.35

109

u/sengirminion 5d ago

Dang. Maybe they ought to hire like 60 thousand employees to help dig through their multi-billion dollar lines of code to prevent this from happening again. Just a thought.

→ More replies (7)

43

u/reversethrust 5d ago

lol. Just read the very terse description. I encountered something similar previously - multiple producer, single consumer problem. The queue worked fine on a laptop, but toss that software on a big server with 256 cores. The test went from taking minutes to taking overnight despite the vast amounts of addition hardware 😂. System utilization on the big machine was much less than 1% though in my case. I wonder what it was like on these exchange servers.

→ More replies (2)

33

u/Nings777 5d ago

It was Clippy with a pitchfork

→ More replies (1)

67

u/Imd1rtybutn0twr0ng 5d ago edited 5d ago

And right below this post, I have an ad for them for AI... irony at its best.

Microsoft, forget AI. Focus on creating workable patches and less bloat. Apologies in thinking you want your consumers to like using your product...

8

u/yinsotheakuma 5d ago

Executive, sincerely: "Where's the profit in consumers liking your product?"

→ More replies (1)

4

u/Sexiarsole 5d ago

But Microsoft didn’t care about workable patches and reducing bloat before AI, why would they stay caring now?

13

u/Anok-Phos 5d ago

Microsoft will never get better and it is up to anyone reading this to ditch them in any and every way possible, even if it's dropping a little bit at a time.

If you work for companies that use Microsoft and have any say, tell your superiors what is going on if they're too dumb to realize it after the latest in ever increasing outages.

Microsoft does not care about their products. You are their products. And yes, I mean they don't care about Windows etc. or you. They care about power and money. You are paying money to be spied on and trapped in their ecosystem, and the enshitification will continue forever.

Leave.

→ More replies (5)
→ More replies (7)

86

u/arekitect 5d ago

Now it all makes sense. Microsoft must have installed Teams on the Exchange servers. Everyone knows Teams can suck the life out of even the most powerful computer. Apparently, it finally found a machine it could bring down faster than my laptop. Unfortunately, that machine was Microsoft's entire cloud infrastructure.

19

u/Earlgrey_fanatic 5d ago

I keep seeing this sentiment being echoed around and it got to the point our "IT consultant" requested we get 32GB of RAM for our next laptop because he said "Teams is power hungry". But I've never actually seen Teams eat that much memory in our laptops. Sometimes it freezes and it's slow, but it's never because of lack of memory.

I genuinely do not understand

14

u/Audisek 5d ago edited 5d ago

The main memory usage is not showing under Microsoft Teams in the task manager, but under msedgewebview2.exe, of which there are multiple processes.

Right now running Teams and Outlook, my work laptop has about 25 of thess processes with over 2 GB committed memory between them.

→ More replies (2)
→ More replies (5)
→ More replies (1)

36

u/Emotional-Inside3126 5d ago

Bug? Or borged patch ?

7

u/Producer_Earth 5d ago

Someagentbrokeinthesystemandexploitedthedayonebug cough cough

6

u/coma24 5d ago

I'd actually love to see this said in person during a debrief, just like you wrote it.

→ More replies (1)

19

u/brattysweat 5d ago

Lucky those planes grounded to get everyone to forget this even happened

54

u/PressureBeautiful515 5d ago edited 5d ago

How many times are we going to read a sentence like "the system had built-in failover for just this eventuality, but unfortunately that just caused the problem to spread and made everything worse" or words to that effect? It seems to be quite a common theme in these postmortems.

Sometimes the problem gets way worse because of how the system is designed to try to reroute and keep working, to be "fault tolerant." Instead it ends up "fault aggravating."

EDIT: diagramofplanewithreddots.jpg

79

u/TA_DR 5d ago

Ask yourself the opposite: how many times are we going to read the sentencd "the system had built-in failover that worked perfectly, therefore nothing happened"?

Probably not so many.

15

u/Un_Ballerina_1952 5d ago

Perfect reply. I worked hard at a large company to ensure that the calendar/clock changes at midnight on 12/31/1999 didn't bring down all our databases and engineering software. The fixes worked (with months of planning, coding, testing, tweaking) and in the end, far too many people were saying, "See; it really wasn't the big deal everyone was trying to scare us about!"

→ More replies (6)

22

u/WestaAlger 5d ago

Probably a self filtering effect though. How many times are people going to report on a bug that WAS mitigated automatically and never had an end user impact?

→ More replies (12)

13

u/NoBonus6969 5d ago

I hate it when bugs don't announce themselves at production

35

u/enthusiastic_duck 5d ago

All eyes are now on Microsoft c-suite who decided to lay off senior engineers in hopes to promote AI. The bug was there since ages, but now they have limited expertise to navigate through such issues

→ More replies (1)

6

u/TyrusX 5d ago

All bugs are hidden. But some are known in advance.

6

u/Iron_Wolf123 5d ago

Let me guess, Win 11 24H2 update?

→ More replies (1)

24

u/SnooOpinions8790 5d ago

In 2014 Microsoft abolished its dedicated test practice

In the following 5 years I slowly removed all Windows computers from my house and moved services to other providers. It was always just a matter of time before this happened.

When OpenAI started looking elsewhere for their cloud last year despite their close contractual ties to Microsoft it was obvious that the Microsoft cloud stuff was teetering on the brink.

I will point out that having completely lost the knowledge and skills to test "black box" software they were then in a particularly awful position to start using AI to generate code that none of their engineers have a deep understanding of. But that may not be the immediate cause of this outage - which is the cumulative crap of a decade of declining software engineering skills and bullshit managers rising to the top because there is no dedicated bullshit-detecting practice left in the company.

19

u/dorkmuncan 5d ago

"The disruption lasted from 4:20 PM UTC on November 12 until 4:10 PM UTC on November 13."

Suspiciously 10 mins under 24 hours.

8

u/costconormcoreslut 5d ago

Why suspiciously? What happens at 24 hours?

37

u/v_neet 5d ago

It becomes Microsoft 364

→ More replies (2)
→ More replies (6)

5

u/Acceptable-Land7827 5d ago

Yeah, so they can round it up to 365

15

u/cornbread6767 5d ago

Good thing you just fired 50,000 devs 🙄

→ More replies (2)

45

u/MothChasingFlame 5d ago • Bravo

So it was AI and bad processes established by managers with no brains. Got it.

15

u/Jon-Umber 5d ago

bad processes established by managers with no brains

This is, overwhelmingly, my experience with all disciplines in corporate America.

Leadership must be damn near impossible to teach, because for every good leader I've worked for, I've had 4 godawful ones.

→ More replies (1)
→ More replies (1)

18

u/PurpleCaterpillar451 5d ago • Bravo

Bugs are always hanging around windows

5

u/SansSariph 5d ago

I've joked before that just like "everyone" should hold a retail job at least once to learn about it, "everyone" should work a week as an on-call engineer for large scale distributed services.

The weird problems with data, load balancing, and the chain reactions you can cause by recovering, the ability to DDoS yourself with your own clients retrying requests, the entire problem space of dealing with a never ending stream of input that you "keep up with" - it's hard. 

Working on services changed my mindset about retrospectives and root causing problems. It's (almost) never just one thing's or one person's "fault" except at a surface level.

25

u/SaveDnet-FRed0 5d ago

TDLR: Microslop's AI generated code base crashed leading to automated backup systems activating that spread the problem to unaffected servers and no one at Microsoft was competent enough to realize what was going on until days after the problem became to big to ignore.

...This also implying that a repeat incident is much more likely to eventually happen then not. Anyone relent on Microsoft should look to switch to an alternative ASAP.

17

u/IncognitoAnonymous2 5d ago

Reports like this are called "damage control"

11

u/Thisappistrash69 5d ago

Linux needs to become a more household name. I think it’s time to move away from windows. We need alternatives

→ More replies (1)

37

u/ZapDapper 5d ago

Hidden just means that they didnt actually look for the case where it happens.

"I closed my eyes and saw nothing"

17

u/mcagent 5d ago

It was live in production for 3 months. It wasn’t an easy thing to find

5

u/mariosunny 5d ago

Oh yea, let's just have QA run load tests on every single piece of unique hardware across our highly heterogenous infrastructure.

→ More replies (2)

4

u/FrankieMint 5d ago

Microsoft 364, 363, 362...

4

u/silverwoodchuck47 5d ago

TLDR: Microsoft tests their software in production.

4

u/Complete_Entry 5d ago

It's always a fuckup update and then they roll back the statement.

10

u/Splendid_Carpark 5d ago

A Microsoft product with a bug? Hard to believe.

12

u/castlite 5d ago

The bug is named Copilot

7

u/romulof 5d ago

This is the first time a bug was hidden. They are usually stamped in the facade by developers.

15

u/KraljZ 5d ago

And yet the bug wasn’t found by AI

11

u/unexpectedreboots 5d ago

Man the comments lol. The change was running in production for over 3 months without issue.

A very specific series of events caused it to be exposed. Read past the headline.

12

u/Hightower840 5d ago

Earlier this year MS said that 20 to 30% of it's code was now AI written...

→ More replies (3)

6

u/buzz_light365 5d ago

Where's the link to Microsoft postmortum RCA? This article doesn't have a source. Is this just guesstimates and assumptions from the author??

6

u/Privateer_Lev_Arris 5d ago

A bug in Microsoft? Can't be. /s

8

u/blur410 5d ago

You would have thought that they could have sent an intern to the Best Buy and picked up a new computer. /s

→ More replies (1)

57

u/Major303 5d ago edited 5d ago

First they claim AI is perfect and flawless and people are obsolete, and now they say something about "hidden" bug. Make it make sense. By their logic AI should catch it effortlessly.

I'm not an average anti AI person, but AI tech bros have abandoned common sense.

27

u/TriLink710 5d ago

I think the real takeaway is that if companies want to get rid of developers, they are going to need a boat load of QA testers. AI does not check its work or truly understand things well enough to perform adequate testing.

And the worst part is, if a bug is found, theres no developer who will have an idea of what could be causing it.

→ More replies (34)

3

u/HotBBQWin 5d ago

Good thing they got us updating every 5 days

3

u/gormami 5d ago

This reads very much like some outages that happened at Verizon Wireless during my tenure there, and is what I expected, honestly. I'm not as familiar with the internals, but it looks like a control plane storm event. Connections cost too much, and then it tips over where the time to service exceeds the timeout, resulting in a state where no real work is being done. The only way to handle it in the end is to shut it down, then bring it back one chunk at a time to let the initialization work get done and settle back down.

This is perhaps the biggest challenge of scale engineering, as there may be dozens to hundreds of different types of activities with their own timeouts and retry algorithms. Aligning all of them properly is very hard, and changes may be made constantly. Each node needs to protect itself in such a way that some real work gets done for each cycle it spends to have any chance of the system recovering itself. Mostly, that means tracking the time spent, and rejecting new requests if the value exceeds the known timeout. Northbound, applications need to back off, rather than accelerate, when undergoing problems, to keep from overwhelming the system. Even then, constructive interference from multiple processes can make it very hard to get it back under control.

3

u/Tupperbaby 5d ago

It's not hidden anymore.

3

u/NapierNoyes 5d ago

Oh, so they basically DDOS’d themselves.

3

u/martixy 5d ago

Ah, it seems to have been an old-school complex system failure and not part of the AI bugpocalypse.

How refreshing.

3

u/Arkaqiu 5d ago

As I understand, race condition on shared resource. Its interesting how choosing proper test harness and environment is crucial.

Butazzo in one of his book shows how one solution is schedulable on 2 cores but it failed on 3 cores. One

3

u/HowsItBeenBen 5d ago

With all the other stuff thats been goin down lately, I call bullshit.

3

u/LocalObligation5642 5d ago

Took down or offered new opportunities to increase capitalization?

3

u/Adderall_Rant 4d ago

I knew it. It was a weather balloon.

3

u/Nosbus 4d ago

Clippy strikes back...

18

u/OffalSmorgasbord 5d ago

So AI wrote the code and AI tested the code and AI monitoring missed the problem and the humans let go for AI to make the shareholders happy weren't there to fix what AI did...

"We will just sue because management didn't disclose how dependent the products are on AI" - Institutional Investors

11

u/CucharitaDePalo 5d ago

Not what the article said, no.

→ More replies (4)
→ More replies (1)

5

u/sadmistersalmon 5d ago

one hidden bug was the root cause. But as you read the rca, you see multiple instances of other “unrelated issues” with corresponding recovery and compensating mechanisms fighting to restore operations. it’s as if you have a heavily leaking bucket that you are trying to keep full by pumping water non stop.

5

u/jcauch 5d ago

I've come to hate this company so much. They've been around for 50 years yet Powerpoint can become unstable when trying to copy paste. They'd never gain market share today with the poor product quality they deliver. Make the simple stuff work. Stop with the slop.

5

u/InAppropriate-meal 5d ago

In other words they pushed some slop-coding and it screwed everything up on an impressively large scale 😂

14

u/FlamingoOk3026 5d ago

Never use windows for mission critical applications.

8

u/soundman32 5d ago

You realise that a huge majority of Microsoft servers are Linux, right?

→ More replies (5)
→ More replies (3)

15

u/Fickle_Competition33 5d ago

The fact the outage was global, not regional, tells a lot how amateur the whole ecosystem is. You can badmouth AWS as much as you want, but their outages and design are always region-bound.

17

u/sessiontoken 5d ago

That's just not true lol. A year ago was the record breaking global AWS outage. https://www.akamai.com/blog/security/when-cloud-breaks-lessons-aws-outage

→ More replies (6)
→ More replies (2)

7

u/msc1 5d ago

"Country Blocked

Access to this page is forbidden."

If you can't deal with script kiddies around the world, maybe don't host cyber security news.

10

u/julioqc 5d ago

so they say... god forbid the tech they invested billions into blindly has not delivered promises

4

u/rigsta 5d ago

Nah I'm sure it was just a new router

4

u/wretch5150 5d ago

Yeah, outsourcing everything including development on windows, then vibe coding m365 updates will do that.

4

u/strcrssd 5d ago

So they developed an internal only patch, deployed it, and it failed. Not mainlined or beta dogfooded, but code that they see as crucial for their operations that they don't ship to customers.

Another reason to stay away from MS. They don't even run their own stock software.

8

u/Sans-valeur 5d ago edited 5d ago

Man when I started investing I thought, it’s Microsoft. How much of the world runs on their products? As safe a bet as any in terms of single stock investments.
I mean, it hasn’t been a bad investment, but yesh, this decade….
The fuck are they doing?
I bought my first Mac ever last year, and I’d never even really considered it before.

This is one of the inherent problems with late stage capitalism.
The goal isn’t to create something good, and then keep it good.
It’s relentless, constant growth by any means necessary, gambling with the entire fucking company if need be.
Constantly looking for the line, how much you can reduce quality while maximizing profit before you drive away your customer base.

5

u/junktech 5d ago

Hey, Microsoft, do you feel like disclosing the backlog? Like the real one with silenced alerts. Not the official one.