r/sysadmin • • 10h ago Regret

PowerCLI multiple default server mode just cost me my job

507 Upvotes

I work (well, worked) at an MSP. One of our bigger clients has a prod vCenter at their main site and a separate vCenter at their DR site. Twice a year we restore their critical VMs from backup into the DR cluster on an isolated network. Their app owners log in and sign off, then we tear it all down. We restore with the original names because a couple of their apps freak out if the hostname changes, and since it's all in a bubble it's never been an issue.
Tuesday was teardown. 31 VMs, and the DR datastore was sitting at like 92%, so I wanted them actually gone from disk, not just pulled from inventory. I had the name list from the restore job, so I ran the same little script I've used after the last three tests:
$vms = Get-VM -Name (Get-Content .\drtest.txt)
$vms | Where-Object PowerState -eq 'PoweredOn' | Stop-VM -Confirm:$false
$vms | Remove-VM -DeletePermanently -Confirm:$false
Yes Stop-VM, not Stop-VMGuest. They're throwaway DR copies, I didn't care about clean shutdowns.
What I forgot is that earlier that morning I'd connected to the prod vCenter in the same PowerShell window to look at a snapshot for a different ticket, and I never disconnected. And at some point years ago I'd said yes to that PowerCLI prompt about working with multiple default servers. So Get-VM without -Server pulls from every vCenter you're connected to.
Same names on both sides. So $vms had 62 objects in it instead of 31, and I never looked.
The power-off step went through all of them. I didn't notice, because a bunch of the DR copies were still running and a wall of power-off output is exactly what I expected to see. Then the deletes started. I'm half watching, and it's still going after it should've been done. About the same time Teams starts going off. I hit ctrl+c, and by then 6 of the prod VMs were deleted from disk: a SQL server, two app servers, an RDS host, their file server, and a print server (the only one nobody missed). The other 25 prod VMs had just been hard powered off in the middle of the workday.
I called my manager right then and told him exactly what I did. There wasn't any point doing anything else, my username is on every event in vCenter.
Most of the 25 came back fine once we turned them on, though one of the other SQL boxes needed some babysitting. Backups were good, thank god. Instant recovery had the 6 deleted VMs running off the backup repo within about an hour, but the repo is a box of spinning disks and SQL running off it was painful. Migrating everything back to real storage wrapped up around midnight. The last backup was 10pm the night before, so the file server lost most of a day of work. SQL had log backups, so it only lost a few minutes.
The client's IT director called our owner Wednesday and said he didn't want me in their environment again. They're about a quarter of our revenue and I was basically the guy on that account. Friday I got the "this isn't working out" call. Honestly can't even be that mad, it was completely avoidable.


r/sysadmin • • 21h ago

Does Microsoft's Entra Connect Health cloud API breach give you pause?

103 Upvotes

As you know, Microsoft's Entra Connect Health cloud API was breached within the past few days.

The breach itself does not seem catastrophic, but the timeline and some details give me some concern.

July 23, 2026: Microsoft introduced vulnerability into API

Nov. 9, 2026: Microsoft detected vulnerability

Nov. 19, 2026: Microsoft deployed fix

Nov. 20, 2026: Microsoft completed investigation and confirmed unauthorized access to tenants

Nov. 21, 2026: Customers notified

With all its AI usage, it took Microsoft 4 months to detect a vulnerability. It took 10 days to stop the bleeding. And why was an expired certificate kept on its server and available via API for two years to begin with? To me, this seems sloppy. And it was allowed to slip through the cracks for two years? Does Microsoft not review its infrastructure?


r/sysadmin • • 12h ago

How to provide email business continuity when primary mail server (Exchange Online) outage occurs? (In relation to MO1550821)

14 Upvotes

Hi All,

After Microsoft incident MO1550821, we are reviewing whether our current email architecture provides enough business continuity.

We have roughly 3,500 users and a hybrid Exchange environment (mainly Exchange online however some "legacy" mailboxes still on-prem). To estimate our business scale - during the approximately 24-hour incident window when the incident occured, our organization generated around 60,000 outbound messages.

We intend to continue using Microsoft 365 as the primary email platform. The question is what should happen when Exchange Online or a related Microsoft dependency is unavailable for an extended period.

  1. Can Exchange on-premises act as a redundant platform for Exchange Online?

Could synchronized copies of Exchange Online mailboxes be maintained on-premises and activated during an outage?

For example, could the normal mailbox use example.com domain, while a synchronized emergency mailbox usesexample.eu? The exact domain is not important. The objective would be to give users access to recent messages and allow them to continue sending and receiving while Exchange Online is unavailable.

Is this technically possible and supported, or would it create problems with synchronization, mail routing, duplicate messages, calendars, compliance and mailbox consistency?

2. Could another email platform provide this redundancy?

If on-premises Exchange cannot do this, could an independent email server or hosted mail platform maintain synchronized standby mailboxes?

What is the current industry-standard alternative: a secondary mail system hosted on the organization’s own servers, another hosted email provider, or a dedicated email continuity service? We are looking for a solution that could provide users with access to recent messages, receive and send email through the corporate domain during an outage, and synchronize the outage-period messages back to Exchange Online after recovery. If you have any vendors, solutions worth checking - please share it with me, I am happy to review (my background is not exactly Email administration, more general organization BCP side).

At the end to add complexity our current mail flow includes a secure email gateway, as well as a separate email DLP layer in the MX/mail-flow path - we would like to have this still be implemetned on the backup mail infrastructure.

Edit:

I know that redundancy costs money but for know I want to know what are the potential technical options which can be implemented to be more resistant.

First recommendation was Mimecast which I wasn't previously familiar with - big thanks I am checking its functionality right now.

I also know that if majority orgs relay on MS and they are down, I won't be able to reach them however still MS doesn't have 100% of the market, there are some companies Google based, or smaller email servers hosting based so not 100% of destinantions would be down.


r/sysadmin • • 4h ago

Sysprep /generalize /oobe vs. post-clone SID changers (SIDCHG / Wittytool) with FOG Project - What are you actually using in production?

13 Upvotes

I manage around 700 Dell OptiPlex workstations at a university (across 2–3 models, including older OptiPlex 3020 and 790 units) using FOG Project as our image deployment server. We standardized on Windows 11 23H2 as our main OS to avoid the heavier background AI overhead in newer builds on our legacy hardware.

Right now, my workflow is building our golden image in Audit Mode, installing all required software and then running sysprep /generalize /oobe /shutdown before capturing the image with FOG. However, I constantly run into AppxPackage validation errors during Sysprep, sometimes over basic built-in apps like Notepad, Calculator, Sticky Notes, or Snipping Tool forcing me to track down and remove packages before Sysprep will complete.

This made me wonder why some admins skip generalizing altogether, capture the live image directly with FOG, and just use a post-clone SID changer (like SIDCHG or Wittytool) alongside the FOG Client to handle ID resets afterward.

I recently re-read Mark Russinovich’s classic post,The Machine SID Duplication Myth and Why Sysprep Matters, which breaks down why duplicate SIDs aren't the real issue, it's all the other machine-specific IDs (WSUS client IDs, KMS CMIDs, endpoint agent IDs, certificates, etc.) that Sysprep clears. (Worth noting too that recent Windows 11 security updates like KB5065426 have started enforcing stricter machine SID uniqueness checks for Kerberos/NTLM authentication, making post-clone ID conflicts an even bigger headache.)

For those managing similar fleets (especially with FOG Project) in production:

  • What does your actual imaging workflow look like, are you strictly running sysprep /generalize before FOG capture, or are any of you capturing non-sysprepped images and using post-clone ID tools without issues?
  • How do you prevent Windows 11’s built-in apps (Notepad, Calculator, Snipping Tool, etc.) from updating or registering per-user in Audit Mode so Sysprep doesn't break? (Note: I already build completely offline with the network cable unplugged and only install standalone manual updates, yet it still triggers per-user Appx errors during Sysprep.)

Appreciate any insight into what’s working cleanly in your environments.


r/sysadmin • • 18h ago

Worried about the future

12 Upvotes

I know this post was probably made before but i wanted to give my two cents. I finished college (which was a lot of maths+computer science)3 years ago and my first job was a standard support l1 +lv2 role it support role. I am now on my second job in the field which is basically a mix of sysadmin and sap 4hana/dm support

I am content with it since the pay ain't bad for my experience and also I feel like I would benefit a lot from learning sap. But lately I keep worring since I really wanted to make a career in this field, but I feel hopless because of AI, yeah. That's it. How do you feel about this?

Edit:Forgot to mention, my biggest concern is the fact that I know that AI can't replace sysadmin right now, but who knows how advanced will it be in 2, 5, 10 years


r/sysadmin • • 6h ago

Question What does good with servers and networking actually look like in your day to day job, and what should I look for on a resume?

11 Upvotes

Hello.
In most IT environments, I think it's rare for roles to be strictly separated into only servers or only networking(maybe...)
You usually end up dealing with both (such as Windows Server 2022, VMware, VLANs, routing, etc.)
Recently, my manager asked me to look through a stack of resumes and pick out candidates who are strong in either servers or networking.
I'm not quite sure what specific keywords, experiences, or skills on a resume prove that someone actually has real competence in those area
Please tell me from the perspective of those working in the field
In day-to-day work, what does being capable in servers and networking specifically look like?
When screening resumes, what specifically do you look for to determine whether someone truly has the knowledge and practical skills?
Thank you.


r/sysadmin • • 15h ago

Question - Solved Root CA cert lands in Intermediate store and Trusted Root after autoenrollment

7 Upvotes

I've just built my first 2-tier PKI and published the root and intermediate certificates to AD with certutil -dspublish so domain joined devices pick them up automatically.

I noticed the root CA cert was getting placed in the Intermediate CA store as well as Trusted Root store on clients.

I read somewhere that autoenrollment copies the AD AIA container into the Intermediate store, and that publishing the root with certutil -dspublish puts a copy in AIA.

I deleted the root’s entry from CN=AIA,CN=Public Key Services,CN=Services,CN=Configuration in ADSI Edit and ran certutil -pulse on a test machine.

The root cert disappeared from the Intermediate store so I thought it was fixed, but I‘ve noticed the next time a device autoenrols for a template, the root is placed back into the Intermediate store.

I know this is probably harmless for chain validation, but I‘m just curious if this is expected behaviour.


r/sysadmin • • 5h ago

Increase in SPAM being allowed through EOP ?

3 Upvotes

Has anyone else noticed a massive increase in the amount of SPAM being allowed through EOP very recently? It seemed to start yesterday, November 21st. Really obvious SPAM is coming through with an SCL of 1. To be clear, I'm not talking about virus laden attachments, but emails in languages other than English, emails with no text (all HTML), etc. The types of emails that would at least not make it to the focused inbox.


r/sysadmin • • 11h ago

General Discussion What’s the actual process for getting the right people on a call when something breaks?

1 Upvotes

Curious how other teams handle the first few minutes of an outage.

When a server, a VPN or a key app goes down, there’s usually a scramble: someone finds the bridge link, someone else works out who’s on call, the manager and the vendor get pinged, and people join one by one while you’re still trying to diagnose.

  1. Do you have a defined process for this, or is it “whoever’s awake starts a Teams call”?
  2. How long does it usually take until the people you need are talking?
  3. What tooling do you use (Teams, Slack huddles, a standing Zoom bridge, PagerDuty/Opsgenie)?

r/sysadmin • • 18h ago

General Discussion Kill Copilot in a work-account licensed macOS Office suite

0 Upvotes

Admittedly written on vibes; but very good ones.

For anyone who finds Microsoft's shameless bombardment of Copilot as offensive and in poor taste, this script will remove Copilot from the ribbons of the latest versions of Word and PowerPoint.

https://gist.github.com/DeathToMediocrity/b91653b018c77d6c24eadf6c78c0571e

Hoping this helps some people find an extra measure of peace and calm they didn't have before.