r/sysadmin
•
u/Apartheid20
•
11h ago
PowerCLI multiple default server mode just cost me my job
I work (well, worked) at an MSP. One of our bigger clients has a prod vCenter at their main site and a separate vCenter at their DR site. Twice a year we restore their critical VMs from backup into the DR cluster on an isolated network. Their app owners log in and sign off, then we tear it all down. We restore with the original names because a couple of their apps freak out if the hostname changes, and since it's all in a bubble it's never been an issue.
Tuesday was teardown. 31 VMs, and the DR datastore was sitting at like 92%, so I wanted them actually gone from disk, not just pulled from inventory. I had the name list from the restore job, so I ran the same little script I've used after the last three tests:
$vms = Get-VM -Name (Get-Content .\drtest.txt)
$vms | Where-Object PowerState -eq 'PoweredOn' | Stop-VM -Confirm:$false
$vms | Remove-VM -DeletePermanently -Confirm:$false
Yes Stop-VM, not Stop-VMGuest. They're throwaway DR copies, I didn't care about clean shutdowns.
What I forgot is that earlier that morning I'd connected to the prod vCenter in the same PowerShell window to look at a snapshot for a different ticket, and I never disconnected. And at some point years ago I'd said yes to that PowerCLI prompt about working with multiple default servers. So Get-VM without -Server pulls from every vCenter you're connected to.
Same names on both sides. So $vms had 62 objects in it instead of 31, and I never looked.
The power-off step went through all of them. I didn't notice, because a bunch of the DR copies were still running and a wall of power-off output is exactly what I expected to see. Then the deletes started. I'm half watching, and it's still going after it should've been done. About the same time Teams starts going off. I hit ctrl+c, and by then 6 of the prod VMs were deleted from disk: a SQL server, two app servers, an RDS host, their file server, and a print server (the only one nobody missed). The other 25 prod VMs had just been hard powered off in the middle of the workday.
I called my manager right then and told him exactly what I did. There wasn't any point doing anything else, my username is on every event in vCenter.
Most of the 25 came back fine once we turned them on, though one of the other SQL boxes needed some babysitting. Backups were good, thank god. Instant recovery had the 6 deleted VMs running off the backup repo within about an hour, but the repo is a box of spinning disks and SQL running off it was painful. Migrating everything back to real storage wrapped up around midnight. The last backup was 10pm the night before, so the file server lost most of a day of work. SQL had log backups, so it only lost a few minutes.
The client's IT director called our owner Wednesday and said he didn't want me in their environment again. They're about a quarter of our revenue and I was basically the guy on that account. Friday I got the "this isn't working out" call. Honestly can't even be that mad, it was completely avoidable.