r/kubernetes • • 21d ago

Periodic Monthly: Who is hiring?

25 Upvotes

This monthly post can be used to share Kubernetes-related job openings within your company. Please include:

  • Name of the company
  • Location requirements (or lack thereof)
  • At least one of: a link to a job posting/application page or contact details

If you are interested in a job, please contact the poster directly.

Common reasons for comment removal:

  • Not meeting the above requirements
  • Recruiter post / recruiter listings
  • Negative, inflammatory, or abrasive tone

r/kubernetes • • 2d ago

Periodic Weekly: Share your victories thread

4 Upvotes

Got something working? Figure something out? Make progress that you are excited about? Share here!


r/kubernetes • • 2h ago

how do you clean up forgotten stuff in a cluster like old namespaces and orphan pvcs

10 Upvotes

feels like clusters slowly fill up with junk nobody owns. old test namespaces, pvcs left behind after a statefulset got deleted, configmaps no one references anymore. do you run some cleanup job, use labels with an owner and expiry, or just do a manual sweep every few months? curious what actually sticks long term


r/kubernetes • • 20h ago

Moved my Kubernetes secrets to Bitwarden with External Secrets Operator

76 Upvotes

My homelab cluster runs from a public Git repo through Argo CD, except Secrets. Those were still kubectl create secret from my laptop.

Now they live in Bitwarden Secrets Manager. External Secrets Operator syncs them into normal Kubernetes Secrets.

Git only holds references, no values. The cluster’s access is read-only, and a rebuild needs one token.

Repo: https://github.com/prateekjaindev/homelab

Write-up: https://blog.prateekjain.dev/kubernetes-secrets-on-a-talos-homelab-with-bitwarden-eb5696dfb3be?source=friends_link&sk=0f6495f99946cd6de38c978dc13a4ad7


r/kubernetes • • 13h ago

Anyone using milvus helm or milvus operator in prod

Thumbnail reddit.com
3 Upvotes

r/kubernetes • • 1d ago

CVE Patching Across Hundreds of Workloads

54 Upvotes

With all the recent CVEs being announced, what's your approach to quickly patching hundreds of container workloads spread across multiple Kubernetes clusters?What were some of the biggest challenges you've run into, and how did you solve them? Also, how do you make sure patching becomes an ongoing process instead of a fire drill every time a critical CVE drops?


r/kubernetes • • 1d ago

Kargo and multiple freight from one warehouse

Thumbnail reddit.com
3 Upvotes

r/kubernetes • • 1d ago

Kubernetes security tools that fix the vulnerable images instead of just flagging them?

14 Upvotes

our cluster scanner lists around 900 cves across 40ish workloads and most of them sit in images we pull through vendor helm charts, so rebuilding them ourselves isnt really an option. i've been reading about rapidfort and chainguard, and i'm not sure yet which of them fits charts we don't own. which kubernetes security tools fix the vulnerable images instead of just flagging them...


r/kubernetes • • 1d ago

Way forward after KTHW?

6 Upvotes

So I set up 4 nodes on my proxmox and jumped straight into the Kubernetes the Hard Way series of labs. Needless to say I came all the way to the end but I don't feel like I learned that much. I know the basics of pubkey encryption and TLS and it felt like a lot of it was just setting up a safe way for the nodes to talk to each other.

I am searching for a deeper understanding and better mental model of the whole system.

Does anyone have any experiences to share or tips on where to go from here to learn it in an effective hands-on fashion?

Not working with kubernetes currently but preparing for an interview.


r/kubernetes • • 2d ago

do you set cpu limits on everything or just requests

33 Upvotes

requests make sense to me but cpu limits seem to cause throttling even when the node is idle. memory limits at least stop one pod from eating the node. curious if most people skip cpu limits on purpose or just use the defaults from a helm chart


r/kubernetes • • 2d ago

How do you structure your Flux repo directories?

12 Upvotes

Just getting started with Flux and trying to settle on a folder layout. For example, I'm thinking of something like:

clusters/
└── home/
├── infrastructure/
│ ├── controllers/
│ └── configs/
└── apps/

Do you split by type like this (controllers/configs), or keep everything for one app in its own folder?

Would love to see how others lay theirs out, and links to repos are very welcome. Thanks!


r/kubernetes • • 3d ago

Kubernetes storage in 2026?

42 Upvotes

Hello,

Currently running 3x master and 3x worker nodes in virtual machines, vanilla kubernetes. These virtual machines are running in a vmware environment.

As you all know, Broadcom is Broadcom. We're planning to move away from vmware under 2027, currently we're evaluating Proxmox and HPE VM Essentials.

But, the K8S cluster is running the vSphere CSI plugin for storage, which works great. Once we start migrating i need to have another storage solution in place. Reading about this there is a shit ton of mixed opinions regarding:

  • Rook Ceph
  • Longhorn
  • Piraeus
  • And more

Most discussions are from 2-3 years ago when platforms like Longhorn still had some issues for some people, a lot of people ran Rook Ceph but it was complex and resource intensive. Some ran pure NFS-servers etc.

So I'm just wondering, what are you all using today? Pros and cons of the solution you're using?


r/kubernetes • • 1d ago

Giving a cloud coding agent cluster access without a kubeconfig

0 Upvotes

A cloud agent (Devin here) runs its changed service in its own VM and tests it against the shared cluster through a routed Signadot environment, with one API key and no kubeconfig: https://www.signadot.com/docs/tutorials/devin-signadot-sandboxes

Full transparency, I work for Signadot.


r/kubernetes • • 2d ago

How to backup and restore PVCs with Plakar's Kubernetes Operator

Thumbnail
youtu.be
7 Upvotes

r/kubernetes • • 3d ago

Keeping things updated is... annoying.

25 Upvotes

Yeah, I know, big surprise. But I just started to put together my entire structure and even added GitOps to keep track and sync of what I run. However, in doing so, I realized that I had no "update strategy" - and so I went looking and found... pain.

For reference, I use Flux for my GitOps, it is immediately installed via k0s' k0sctl so that it is available as soon as possible. And that mostly just works. Once Flux is alive, I can apply a GitRepo + Kustomization that points at the root of it all and it just goes. That part is solved and works nice. But of course, things get updated.

I use this structure in my repo:

- docker/...
- opentofu/...
- kubernetes/
  - root.yaml # GitRepo + Kustomization
  - kustomization.yaml # Points at each <component>/flux.yaml
  - <app or component name>/
    - flux.yaml # Tells flux where this component's kustomization.yaml is
    - kustomization.yaml # Actually lists all related files
  - <other thing>/
    - install/... # HelmRepo + HelmRelease
    - config/... # CRD based objects - be that a config or something else
    - flux.yaml # Is a "---" separated YAML for both with dependency tracking

This structure is easy to maintain - although it has an absolute number of repeatance but... its okay. It does bother me but realisticaly, this is just a homelab doing homelab things, I can copy and paste a bit.

The real problem is trying to receive updates. Right now, I bend updatecli in a cronjob to my will using updatecli compose. But this falls apart quickly because the policies for patch, minor and major updates fight each other - a lot. And it falls apart on image tags like Vaultwarden's 1.37.5-alpine; the -alpine tag throws off the comparison and borks the check completely.

So... what can I use to help me here? What do you use to keep track of updates across helm charts and plain container images?

I looked into Kargo but that would instantly explode complexity by a lot - especially since I want a write-back to Git to track what happened when and by whom. (I do use a PAT right now, but I plan on using a GH app in the future for better separation.)

Any recommendations or tips...? Thanks!


r/kubernetes • • 3d ago

Periodic Weekly: This Week I Learned (TWIL?) thread

5 Upvotes

Did you learn something new this week? Share here!


r/kubernetes • • 4d ago

I built a game to learn kubernetes - Night Shift

Thumbnail play.snowops.net
246 Upvotes

I have been working on multiple projects and one thing I felt was the biggest challenge for adoption. Setup time. If something takes up more than 5 minutes, you lose 90% of your potential users.

For this, i created a browser game which helps you learn real world kubernetes incidents within your browser. It's free and no setup required.

All the results are from logs of my another project - Snowops Labs. So, everything you will get is real.

It teaches you how the real engineering team works and gives you real incidents to solve and learn.

I'd genuinely like to hear what experienced engineers think. Especially if you've spent time troubleshooting production incidents.

TL;DR: An attempt to make practical DevOps troubleshooting more interactive and game-like. Give it a try and tell me what you think.


r/kubernetes • • 3d ago

Made a kubectl plugin that finds pods exposed to java4shell

18 Upvotes

java4shell (CVE-2026-114938) only triggers if the node CPU has AVX-512 VBMI, the JDK is affected, and -XX:-UseBASE64Intrinsics isn't set.

For those who don't have security scanners I wrote a small kubectl plugin. It reads the NFD CPU labels and pod specs, optionally execs `java -version` for exact versions and flags, and also warns about pods that could get rescheduled onto a VBMI node.

https://github.com/simonliska/java4shell-k8s-scanner

Any feedback appreciated! (PS: built with help from Claude).


r/kubernetes • • 3d ago

Pod ran out of PIDs, every subsystem failed with a different error, and the liveness probe kept passing

25 Upvotes

Writing this up because the symptoms looked like four unrelated outages and the actual cause was one number.

We run a service in a pod that shells out a lot (az, kubectl, some bash pipelines). One morning it started throwing errors from everywhere at once:

  • DB client: connection reset by peer
  • shell runner: fork/exec /usr/bin/bash: resource temporarily unavailable
  • a fetcher: “config not initialized”
  • a cloud integration marked INVALID with az login failed and 500+ retries behind it

Different errors, different components, same minute. And the liveness probe was green the whole time, so nothing restarted it.

resource temporarily unavailable is EAGAIN. On fork it means the kernel wouldn’t give the container another PID. The pod had hit its pids cgroup limit. Everything that needed to spawn a process was failing, each in its own words. The health endpoint is a plain HTTP handler that never forks, so it answered 200 right through the outage.

What was eating the PIDs: subprocesses that never exited. Some execs had no timeout at all. The one that did have a timeout killed the bash -c wrapper on expiry, but not the az and kubectl it had started, which got reparented and kept running. kubectl in particular was hanging forever on auth for clusters where our service principal was missing a role. A few leaked per scan, a scan every hour, for weeks.

How we confirmed it from inside the pod:

cat /sys/fs/cgroup/pids.current /sys/fs/cgroup/pids.max # cgroup v2

cat /sys/fs/cgroup/pids/pids.current /sys/fs/cgroup/pids/pids.max # cgroup v1

ps -eo pid,etime,cmd | sort -k2 -r | head # oldest processes first

current was at max, and the oldest entries were weeks-old kubectl processes.

What we changed:

  1. Every exec gets a context with a deadline, and cancellation kills the whole process group, not just the direct child (in Go: Setpgid on the child, signal the negative pid). This was the actual bug.
  2. Explicit podPidsLimit on the kubelet (--pod-max-pids), low enough that a leaking pod dies before the node does. We were on the default, which is high enough to take the node down first.
  3. Alert on container_processes from cAdvisor trending up against the limit. It had been climbing in a straight line for weeks and nobody was looking at it.
  4. Considering a liveness probe that actually forks something, since an HTTP 200 told us nothing here.

The lesson I keep coming back to: when several unrelated things break in the same minute, stop reading the individual errors and look for what they share. Here it was the kernel saying no to fork.

Curious what people use for the liveness side. Is “probe that execs something” a thing, or do you just alert on the pids metric and leave liveness alone?


r/kubernetes • • 4d ago

Moved my homelab to Argo CD

Post image
533 Upvotes

My homelab used to run on helm installs from my laptop, and the repo kept drifting from the cluster.

Now Argo CD syncs everything from Git. If you change something directly, it gets reverted. Nothing was reinstalled, it took over what was running.

A few things stay outside on purpose: Cilium, since Argo CD runs on its network, and Secrets, since the repo is public.

Repo: https://github.com/prateekjaindev/homelab

Detailed write-up: https://blog.prateekjain.dev/gitops-for-a-3-node-talos-kubernetes-homelab-with-argo-cd-336c076e93cc?source=friends_link&sk=ebda84ecdc3e2458fead2638374be51e


r/kubernetes • • 4d ago

Periodic Weekly: Show off your new tools and projects thread

2 Upvotes

Share any new Kubernetes tools, UIs, or related projects!


r/kubernetes • • 3d ago

How do you diagnose drift when desired, observed and last-applied state disagree?

0 Upvotes

Disclosure: I work on Zen Mesh; this is an engineering question from our work on distributed reconciliation.

Consider a remote workload where the desired generation is 8, the runtime reports 7, and the reconciler records 6 as the last generation it successfully applied. That gap alone doesn’t tell us whether an observation is delayed, a controller restarted before recording the result, or something else changed the runtime.

We’re trying to keep those three facts separate, with clear ownership, rather than collapsing them into one “drift” flag. A related case: a plan is computed for a remote session, the session disconnects and reconnects, and the old plan resumes even though the desired object hasn’t changed. The desired revision alone may not tell us the plan is stale.

For those operating controllers or GitOps across clusters: what information has actually helped you distinguish delayed observation from an out-of-band change? Do you record last-applied state separately, and how do you prevent an old plan from acting after a reconnect?


r/kubernetes • • 4d ago

EKS 1.31 extended support ends Nov 26, then AWS auto-upgrades the control plane. 1.34 standard support ends Dec 2

8 Upvotes

Disclosure: I'm the founder of Expiry Radar, a commercial tool that tracks dates like these. Everything below works only with aws cli.

Two EKS dates coming up:

- 1.31: extended support ends Nov 26, 2026. After that AWS upgrades the control plane to the next version on its own schedule, so add-ons and node groups that aren't ready can break.

- 1.34: standard support ends Dec 2, 2026. From then on the cluster bills extended support, $0.60 per cluster-hour instead of $0.10 (about $365/month more per cluster).

Check your clusters and what blocks the upgrade (per region):

for c in $(aws eks list-clusters --query 'clusters[]' --output text); do echo "$c $(aws eks describe-cluster --name "$c" --query cluster.version --output text)"; done

aws eks list-insights --cluster-name <name>

aws eks describe-cluster-versions --query 'clusterVersions[].[clusterVersion,endOfStandardSupportDate,endOfExtendedSupportDate]' --output table

`list-insights` is the useful one: it lists deprecated APIs still in use and add-on or node version skew before you upgrade.

Source for the dates: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

How are you handling upgrade from 1.31 to 1.32, in place or blue/green clusters?


r/kubernetes • • 4d ago

Kubernetes Podcast episode 273: Pod Autoscaling and Resource Management, with Adrian Moisey and Jakub Pawełczak

Post image
8 Upvotes

Check out our latest episode: Pod Autoscaling and Resource Management, with Adrian Moisey and Jakub Pawełczak!

https://kubernetespodcast.com/episode/273-pod-autoscaling/