r/devops • • 4h ago

Architecture How do you protect the first entry point from traffic overload?

15 Upvotes

Regardless of whether it's a CDN, WAF, Load Balancer, API Gateway, or Ingress, the first layer receiving traffic has finite resources.

Even with rate limiting, incoming requests still reach that layer. With millions of requests or large payloads (5–10 MB), how do we prevent the entry point itself from becoming the bottleneck?

- Can we reject requests before receiving their entire payload?

- Where should admission control actually happen?

- How do large-scale systems handle this without endlessly scaling the front layer?

Would love to hear real-world architectural approaches and trade-offs.


r/devops • • 1d ago

Discussion Which Day of week do you generally feel safe for prod deployment ? Have you done this anytime on friday ?

Post image
581 Upvotes

An planning release cycle for dev stg and prs. Which day genrally you feel is right ? . Also th3 branching stategies you folow this time ?


r/devops • • 14m ago

Discussion Can a BCA graduate get a DevOps job? Need advice

0 Upvotes

Hi everyone,

I completed my BCA 2 years ago and I'm currently doing a DevOps course that ends in mid January. I have completed AWS, but I don't have any work experience yet.

I have a few doubts and would like to hear from people already in the field:

  1. Do companies hire only BTech candidates for DevOps roles? Can a BCA graduate get in?
  2. Can a fresher get into DevOps directly, or do companies usually want some experience first?
  3. What are the job opportunities for freshers? Should I apply for DevOps roles directly, or start with support, sysadmin or cloud roles and move up?
  4. Should I finish my course first, or start applying and doing internships now?

I'm based in Chennai, so any advice on the job market here would help too. I'd really appreciate honest advice, even if it's tough to hear.

Thanks in advance!


r/devops • • 18h ago

Architecture For those who have gone multi-cloud, how much do concepts carry over from one provider to another?

14 Upvotes

I am thinking about multi-cloud configuration and have a few questions. How many of you run on both GCP and AWS, for example? What does that crossover look like? Are you running Cloud Interconnect, and if so, how much complexity is involved for your deployment? Are you running Kubernetes on multiple cloud providers, and if so, what does that look like? Also curious about how you manage your Terraform deployments and CICD pipelines.


r/devops • • 20h ago

Discussion First time supporting a pentest scoping meeting as a DevOps engineer. What should I prepare?

3 Upvotes

I’m a DevOps/Integration Engineer helping with a web application release. We’re introducing a new hostname/channel for an existing application and our security team wants to determine whether a pentest is required.
Architecture is roughly:
Browser → Akamai → AWS-hosted frontend → API Gateway/backend APIs
I’m planning to bring the UAT URLs, origin details, Akamai configuration, architecture/integration flow, authentication details and relevant PRs/code changes. We’ll also have people from the application and Akamai sides involved.
For those who’ve done these reviews before:
1. What else should I prepare?
2. What will the pentest/security team typically want to know?
3. What should I personally own as the DevOps/Integration Engineer?
4. What are the common things engineers miss when preparing for these discussions?
Any real-world advice would be appreciated.


r/devops • • 11h ago

Ops / Incidents How do you get responders, support and execs talking during a production incident?

0 Upvotes

Not asking about alerting or runbooks. I mean the human side once the page fires.

At 3 a.m. the on-call engineer is paged, then they need to pull in a second engineer, maybe support or the CTO, and someone has to open a call and send the link. Sometimes it takes five minutes before everyone’s in the same room, and the incident is already running.

I’m researching this problem and want to hear from people who’ve lived it. Specific stories help most.


r/devops • • 2d ago

Architecture Should ArgoCD ever be used to "gate" syncs in any enviroment that isn't dev?

54 Upvotes

My current team uses ArgoCD with autosync permanently disabled for anything that isn't DEV, and it is also disabled for anything infrastructure related (e.g. Kafka, otel, ...).

ArgoCD has basically become a "gate" for updates. Moreover, an OutOfSync in ArgoCD doesn't mean anything anymore since it is now normal to have those.

My personal opinion is that this is an anti-pattern. Moreover, I would argue that you are not really doing GitOps if you handle stuff this way. The state on Git may or may not represent the state in the cluster, but you have no way to know that unless you check the actual state (e.g. if ArgoCD app is OutOfSync).

What's your thoughts on this?


r/devops • • 2d ago

Ops / Incidents How to deal with AI slop produced by other people working on a project

93 Upvotes

Spoiler: rant below

I'm working on a shared project of another company. Basically consultancy, but I'm not physically there, and I also some stuff for the company I'm hired in when needed.

On this project, the infra team (~20 people) are full on convinced that AI is great and anyone that writes anything by hand is a complete idiot.

On the other hand there's me, former pentester (yeah, horrible choice to change career looking at the past couple months lol), strongly believing that, while AI is indeed useful, it must not be used to basically write entire Helm charts and stuff.

The current situation is embarrassing, and no one seems to be able to either understand or care about it. Helm charts are never being reused, instead they are written from scratch every time (except external ones).

For example, every single time a PSQL database is needed, an entire new template is created, with its own values, no backups or WAL archiving, no alerting if WAL replication stop, and different structure of values and templates. There's more than 30 different templates by now, and when asking whether it would be fine to attempt a migration of these to a ce

Security-wise, there's more than half a million CVEs in scanned images. No filtering on EPSS/KEV, no patching, no will to use secure base images, nothing.

This project has been running for more than 3 years, how the fuck do you even get to this point with such basic stuff missing? I don't get it.


r/devops • • 2d ago

Discussion Mid-Level DevOps Engineer expectations

18 Upvotes

I'm a Junior DevOps Engineer with 4 YOE straight out of uni and I would like to know from all the Leads/Managers/Seniors etc what do you guys expect from a mid-level engineer compared to junior.

I will list some of my responsibilities to paint a better picture:

- Maintain and update our internal site (Flask, SQL + MSSQL, Apache, nginx, JS)

- Create and maintain on-prem and in-cloud VMs (Azure and vSphere)

- Setup and maintain on-prem AI servers that we used to rollout to the rest of the engineering department (Nvidia DGX boxes with vLLM + load balancer + harness)

- Troubleshooting tickets from other departments within the engineering department and helping them out with their issues

- DevSecOps for CI/CD pipelines where we implement code analysis (SonarQube) and dependency/base-image analysis (Trivy)

- Containerization (docker and k8s)

- setup and configure and maintain internal repo servers via Nexus Sonatype

- Experience in Platform Engineering, built an in-house tool that allows the devs to push a list containing their dependencies which the tool will download and pushes those packages to the internal repos which then can be used by our Jenkins pipelines (all our pipelines are air-gapped)

- Networking (internal/external DNS, configuring network settings for servers/VMs)

- Issuing self-signed certs

- Maintaining a project that is integrated with Zoho which is used to create customer's sales orders and confirmation emails which I work closely with the support department

These are some of my responsibilities and the point of this post is that I am genuinely interested in what others who are above me look for a mid-level engineer. I have already asked the expectations of the company but I would like to know the expectations of the industry in general. Thanks in-advance


r/devops • • 2d ago

Vendor / market research Remote-US DevOps roles sitting open 2-3 months longer than hybrid ones — anyone else seeing this?

26 Upvotes

Been pulling DevOps/platform postings across a batch of company boards for a side project, and one pattern keeps showing up: roles that say remote-US stay open noticeably longer than hybrid or onsite versions of the same role, even when the stack itself (Terraform, K8s, CI/CD) is pretty standard.

If you’re the one doing the hiring, what’s actually slowing it down — comp expectations, time zone overlap, something else? Curious whether this matches what you’re seeing or if it’s just the boards I’m looking at.


r/devops • • 2d ago

Discussion CAB Release Process

5 Upvotes

Has anyone led a CAB release process as a DevOps engineer and willing to share their experiences?


r/devops • • 3d ago

Ops / Incidents nginx2root - I wish you all the best of luck

269 Upvotes

Colleagues, I wish you all the best of luck!!

I know it's been hard lately but we'll get through it! ❤️


r/devops • • 3d ago

Security Upcoming Critical PHP Security Release scheduled for Thursday, 19 November 2026, between 14:00 and 14:30 UTC.

26 Upvotes

r/devops • • 3d ago

Ops / Incidents Practicing AWS projects

10 Upvotes

My AWS Free Tier period is about to expire, and I'm looking for ways to continue learning AWS and building hands-on Cloud/DevOps projects without spending too much money.

What alternatives or solutions would you recommend? Are there any free platforms, sandbox environments, or tools that allow me to keep practicing real-world AWS projects?

I'd really appreciate your advice and suggestions. Thanks in advance!


r/devops • • 3d ago

Security Mass GitHub signout

75 Upvotes

A majority of our users were signed out of GitHub today and had to sign back in. I haven't seen any issues related. Was there some sort of security concern that I missed?


r/devops • • 3d ago

Discussion Is anyone grouping error logs by trace instead of by message? Looking for opinions on an idea

13 Upvotes

Not selling anything and no link here, I’m just looking for opinions before I go further.

First question: does Sentry, Datadog or a Grafana alert already cover this well enough that I’m reinventing the wheel? If so, tell me what it does better and I’ll stop.

The problem: in my setup, an error shows up as thousands of log lines that are really a handful of distinct bugs. Finding the cause means grouping them, then opening the trace for one occurrence and reading a very large OTLP JSON.

The idea:

  1. Pull error logs from Loki
  2. Group them by a fingerprint of the normalized message (IDs, numbers and quoted values replaced by placeholders), so one bug = one group
  3. Take the trace_id that OTel puts on the log and fetch the trace from Tempo
  4. Condense the OTLP JSON into a short summary: spans in order, failed spans marked, exception messages, only the useful attributes
  5. Open one issue per group with count, first/last seen and that trace summary
  6. Later: have an agent read the group and trace summary and suggest a fix

It’s early and only partly built. Not done yet: avoiding duplicate issues across runs, redacting sensitive data, and the fix agent.

Other things I’d like to know:

  • Do your logs reliably carry trace_id/span_id? The whole approach depends on it.
  • Is grouping by normalized message good enough, or would you group by exception type, stack or failing span?
  • If an agent suggested fixes from a trace summary, what would you need to see before trusting it?

Happy to share the repo if anyone is curious.


r/devops • • 3d ago

Discussion Is anyone actually running agents in their deploy or on-call pipeline?

5 Upvotes

I have close to a decade of experience in big tech, and I'm now building a platform to automate the SDLC. Trying to understand what's working in practice on the ops side.

  1. Deployment: any agent-driven canary or rollback decisions in production, or still human-approved pipelines?
  2. On-call: anyone using an agent for alert triage, first-pass debugging or runbook execution? What false positive rate do you see?
  3. Monitoring: anything beyond anomaly detection that actually reduces pages?

What has held up in production and what got ripped out?


r/devops • • 4d ago

Discussion GitHub vs azure DevOps what's your opinion?

34 Upvotes

Currently in my company we run both for different historical reasons and mostly using GH for new projects since it seems the correct path for the future, but it seems that ADO is also being added with new functionalities such as advanced security and yesterday GitHub copilot app support...

Do you think Ado will remain be maintained and new features arriving? Before I thought it will be deprecated sooner or later but now I'm not that sure

What do you think? And what key differenciators do you see?


r/devops • • 2d ago

AI content Claude bestfriend

0 Upvotes

I know AI is a subject of much debate.

But when you get alerts for Nginx, like today, and you have to patch 4 clusters and nearly 300 services, Claude is actually really handy ^^

And honestly, to be fair, I’m not going to turn my nose up at saving several hours of work xD


r/devops • • 4d ago

Discussion Moved my homelab to Argo CD

Post image
313 Upvotes

My homelab used to run on helm installs from my laptop, and the repo kept drifting from the cluster.

Now Argo CD syncs everything from Git. If you change something directly, it gets reverted. Nothing was reinstalled, it took over what was running.

A few things stay outside on purpose: Cilium, since Argo CD runs on its network, and Secrets, since the repo is public.

Repo: https://github.com/prateekjaindev/homelab

Detailed write-up: https://medium.com/@prateekjain.dev/gitops-for-a-3-node-talos-kubernetes-homelab-with-argo-cd-336c076e93cc?sk=ebda84ecdc3e2458fead2638374be51e


r/devops • • 3d ago

Discussion Main use-case of cloud sandboxes and how it works?

0 Upvotes

Hey guys, I've been seeing a shift in cloud sandboxes being mentioned a lot lately in agent architecture discussions. I'm trying to understand the difference in them compared to say, just using either a standard Docker container or a basic cloud VM.

My very basic level understanding is that these cloud platforms give agents isolated environments to run code without using your local machine. Additionally I've seen a post on Ramp Inspect where they boot a separate remote sandbox for coding sessions with pre-warmed repo snapshots. However, I still don't see what they do that regular Docker containers don't.

Feel free to correct any misconceptions I have here lol, just trying to make sure how it all works. Thanks!


r/devops • • 3d ago

Security Do you consider giving your AI agent full access on your machine good OpSec?

0 Upvotes

Does your agent (Claude, Opencode, Codex, whatever) have access to your entire home directory? Can it use your ssh key, git credentials, docker credentials, cloud credentials etc.?

I have always considered these tools dangerous, because they could be following some random instruction from somewhere, so I have only been running them in sandboxed environments (vm's or containers) with their own readonly credentials.

However, this approach seems to be "holding me back" so to speak. I see colleagues who just YOLO their agent and are getting very good results. Agents that run commands on k8s clusters or remote servers through ssh, read logs and code, and can come up with solutions to problems very quickly.

So: I'm feeling conflicted. Keep the security paranoia or go with the flow and see what breaks?

What do you guys do?


r/devops • • 5d ago

Discussion Do you think AI is killing automation in the old way?

25 Upvotes

So, I join a company that supposed to be very "DevOps", but I figure out they are "Dev" and "Ops" separate.
Not to complaint, since many company still worked that way, in the paper they get ISO 27001 and SOC for that workflow. In the mental CEO and managers think they have some one accountable and responsible for production issues (but well they are not...anyway).
Back to the main point, I observe that, most of automation motivation gone, the Ops guys, which basically have a lot of permission and privilege can now just prompt AI to do something...faster than they do it manually, faster than writing an IaC pipeline (some guys already lost the programming skill long before AI era haha).
I'm fighting for the automation right =]] (Acutally I just asking for a gitlab runner so I can run some daily automation haha)... but thinking again, maybe they have a point...prompting the AI is a form of automation already.


r/devops • • 5d ago

Discussion How do you guys discover and manage tools for improving your GitHub Actions workflows?

11 Upvotes

Hi everyone,
I’m exploring an idea for a small tool that analyses GitHub Actions workflows and helps developers identify wasted CI minutes, unnecessary workflow runs, slow builds and potential cost savings.
Before building anything, I want to understand how developers actually deal with these problems.
A few questions:
How do you currently monitor your GitHub Actions usage and costs? Do you actively look for ways to reduce CI/CD costs or only investigate when bills become too high?
Where do you usually discover new developer tools? Reddit, GitHub Marketplace, X, Product Hunt or somewhere else?
Would you consider paying for a tool that automatically identifies workflow waste and suggests improvements?
I’m still exploring the idea, so honest feedback would be appreciated. I’m particularly interested in how you discover and choose tools for your development workflow.
Thanks!