r/linuxadmin • u/Unhappy_Objective845 • 2h ago
r/linuxadmin • u/Moze_mpk • 2h ago
Memory admission, disk cleanup and sleep for a shared build host: lessons from running coding agents on spare machines
I run a small open source tool, boxd, that sends builds from coding agents to spare Linux machines (rootless Podman containers or small virtual machines, one per git worktree). Three host-side mechanisms, and what I got wrong first.
Admission by memory
- All boxes live in one control group with
memory.maxandmemory.swap.max(defaults 85 and 25 percent of host memory), plus per-box limits. - A new machine is admitted only if measured use plus its size fits the cap, and the sizes of running machines plus its size fit the cap too. Measuring alone fails: a young machine has not claimed its memory yet, so twelve 4 GB machines once got in on a 28 GB cap.
- Otherwise: exit 75 and a message with the numbers, for example
no room for another virtual machine (12288 MB of running machines + 4096 MB used by container boxes + 12288 MB asked > 24576 MB, 100% of the 24576 MB cap); wait for one to end or stop one. With--waitthe client asks every 30 seconds and callers are served in the order they first asked. Without it, the client tries the next host. - Lesson: swap on zram is RAM. My count missed swapped memory. A group read 23713 MB with 7.7 GB more in swap and kept admitting machines until the kernel killed an 8 GB one. Now
memory.swap.currentis added, and half the swap the group may use on zram comes off the default cap. - Each machine records its real peak need, and removal says so:
it needed 2900 MB of its 8192 MB at most.
The work area and its cleanup
- It is its own btrfs subvolume, so system snapshots do not keep build data.
- Whole box folders go, never single files: unused for
BOXD_KEEP_DAYS(14), then the least recently used one per pass once the disk passesBOXD_EVICT_AT(70 percent) until it is underBOXD_EVICT_TO(60). The folder of a deleted worktree goes when the client next notices. - On a disk shared with the system, setup picks lower levels: current fill plus 20 points (at most 70), 10 below that, and 3 days.
- Lesson: never judge a cache file's age by modification time. Package tools keep the upstream's old times on unpacked files, so one release deleted live cargo, Maven and npm cache files.
boxd-resetswaps in a clean subvolume in seconds. It refuses while anyone has an ssh session or holdsboxd-keepawake.
A host that sleeps
The optional power module (boxd-idle) suspends after 20 quiet minutes: no ssh connections, load under 0.5, no boxd-keepawake lock. Before sleeping it arms wake on network and, if the disk is at BOXD_RESET_AT (85) percent, runs boxd-reset. The client sends a wake packet before each ssh; the wait is about 15 seconds.
Honest notes
All code was written by Claude Code, an AI coding agent, with me directing and testing on real hosts. It has been public for a few days. Host packages exist for Void Linux and Debian 13 only. MIT license: https://github.com/gsalucci/boxd
How do you handle memory admission on a host with zram swap, and what cleanup budget do you give build data on a disk shared with the system?
r/linuxadmin • u/Adventurous_Rope4025 • 19h ago
SSHole is in KEV now, how are you finding musl/Alpine sshd across your environment?
In case anyone missed it, the OpenSSH password bypass people are calling SSHole got added to CISA’s KEV list this week. It only affects musl-based systems with PAM off and password auth on, so mostly Alpine, plus OpenWrt and some Yocto/Buildroot builds. Glibc distros are fine. There’s a fixed release and distro backports, and turning Password Authentication off is a decent stopgap once you’ve checked key login still works. Hunting for it isn’t too bad. Look for password logins on accounts that should never take one, new authorized_keys, weird files in /opt or init.d. Finding everything that’s in scope is the harder bit for me. For VMs I’m going off Tenable results, and the axonius pilot we have running puts the OS and openssh version from Tenable and the EDR on each host. I’m a lot less sure about container base images and CI runners, nothing we’ve connected to it reports those. How are you inventorying this? Are people actually finding openssh-server without PAM on production Alpine or OpenWrt, or is it mostly lab, CI and cloud images?
r/linuxadmin • u/Potential-Access-595 • 1d ago
NetWatch 1.0: terminal network monitor with a stable CLI/JSON contract and ISP outage reports
NetWatch 1.0 is out. It's an MIT-licensed network monitor in Rust that shows connections, bandwidth, latency and decoded packets in one terminal. One binary, no config needed.
New since 0.40:
- ISP outage reports.
netwatch uptime --since 30d --md > outages.mdsplits outages into router, internet and DNS. The internet counts as down only when 1.1.1.1 and 8.8.8.8 both stop answering. - Diagnose. Names the likely cause of a network problem and the step to run. Five rules are stable, the rest are beta.
- Command palette.
:opens it.Escsteps back out of a drill-down. - Fits 80 columns. Tables drop columns instead of clipping.
- Packet filters take CIDR ranges and
!=. - Headless mode.
netwatch daemon --metricsserves Prometheus metrics.
Subcommands, long flags, config keys, JSON outputs and Prometheus metric names won't break before 2.0, and no 2.0 is planned. The stability policy lists what's covered.
Runs on Linux, macOS, Windows (needs Npcap) and FreeBSD.
brew install netwatch
scoop install netwatch
cargo install netwatch-tui --lockedMore install options in the README. Binaries on the release page. Changelog.
Thanks to everyone who sent patches, packaged it or reported bugs. If something breaks, open an issue.
r/linuxadmin • u/unknownman652 • 2d ago
Minimus alternatives for anyone still running Minimus images past the Oct 22 shutdown date?
We have 14 services still running Minimus images we pulled before Oct 22. They work, but no updated images are coming, and the scan count on those images has crept up every week since the maintenance period ended.
The shortlist so far is RapidFort and Chainguard. What I care about most is how much each one makes us touch per service, since 9 of the 14 have our own layers built on top of the Minimus base. For anyone who has already moved off, which of the Minimus alternatives was the least painful swap?
r/linuxadmin • u/Runnergeek • 2d ago
Can we make a rule against posting AI slop tools?
There is an influx of users posting AI slop apps/tools. Many of these don’t do anything new nor do they solve problems in a novel way. There is also security issues with these. I would say at best most the authors done actually understand Linux at least not in depth but some might be malicious.
No one in a professional capacity (which this sub is centered around) should be using any of these tools.
I propose a rule to ban such posts.
r/linuxadmin • u/xint_official • 3d ago
Immediately patch nginx servers
Critical vulnerability nginx2root went live today shared by Xint Research.
A Xint researcher found the specific vuln that hits nginx in its default config. This discovery was around the same time he found another critical vulnerability in a different project that we cannot talk about yet due to the same concerns as nginx2root. We took those learnings and incorporated them into the harness as a new, general Xint feature (that is, not fine-tuned to nginx), which then found the other 4 critical nginx vulnerabilities with no human guidance.
r/linuxadmin • u/8lue7or • 3d ago
If you moved from ISC DHCP to Kea, how did you test it before cutover?
I’m exploring a small paid CLI that would compare what the old DHCP server and Kea offered to the same clients during a test. Before writing it, I’d like to understand how people already check this in practice.
If you’ve done this migration at work, what did you actually test before switching over? Did you capture replies from both servers, check a few representative devices, or use some other approach?
If you did compare captures, which took more time: getting usable captures or comparing the replies? I’d also love to hear if a quick check or a script was enough for your setup.
r/linuxadmin • u/ryanmerket • 3d ago
NGINX patches a critical HTTP/2 flaw that Ikotas Labs calls NGINX2Shell
runtimewire.comr/linuxadmin • u/Internal-Cellist-755 • 2d ago
Linux Perf Doctor - Read-only Linux performance diagnostics for AI agents and SRE workflows
Full disclosure: I'm the developer behind this project.
I've been building an open-source project called Linux Perf Doctor.
The idea is to help AI coding assistants and SRE workflows investigate Linux performance problems using actual operating-system telemetry instead of immediately guessing at application-level causes.
Linux Perf Doctor is an Agent Skill that collects and correlates information from Linux interfaces such as /proc, /sys/fs/cgroup, and Pressure Stall Information (PSI).
What it currently supports:
- Host CPU, memory, and system telemetry
- Process-level CPU, RSS, swap, page faults, and context switches
- Linux cgroup v1/v2 analysis
- CPU quota throttling and memory events
- Network socket and block I/O diagnostics
- Optional capability detection for tools such as
perf,strace, andiostat - JSON and Markdown diagnostic reports
One thing I focused on was safety:
The default collection workflow is read-only. It does not invoke sudo or modify kernel parameters. Active profiling with perf and strace requires explicit opt-in and bounded execution.
The project is currently at v1.0.0.
Try it:
npx skills add AnujTiwari-Student/linux-perf-doctor --skill linux-perf-doctorGitHub:
https://github.com/AnujTiwari-Student/linux-perf-doctor
Demo:
https://lnkd.in/p/dRXQRwFk
I'd appreciate feedback from people who work with Linux, containers, DevOps, or performance engineering.
A few things I'd particularly like feedback on:
- What additional Linux performance metrics would you find useful?
- Are the current diagnostic findings and reports actionable enough?
- What would make this more useful for investigating real container or process-level performance problems?
Thanks for checking it out!
r/linuxadmin • u/LengthEducational238 • 2d ago
Golim - an assistant designed for sysadmin work and OS operation
galleryI have been using Linux semi-exclusively for 3 years now and know my way around it well, but I was tired of having to fumble around with configuration files for VPNs, automations, networking setups, navigating deployed machines through SSH as well as the long troubleshooting sessions when something stops working, packages break or a kernel update breaking things.
That's why I made Golim
Golim is an AI agent that's designed from the ground up to operate unix-based machines, figure out the environment quickly and do what you ask it to, efficiently and transparently.
Compared to other AI Agents, Golim has:
- Simpler UI, all actions are observable with support for terminal streaming updates
- a specialized toolset for the job
- first-class support for local inference like Ollama or llama.cpp
- a highly context efficient runtime to minimize hallucinations and get better use out of local models.
Golim can also be lent your sudo privileges on an instant-by-instant basis without ever saving your password, allowing it to complete a much larger range of tasks.
Lastly, it comes with a headless mode so that requests can be automated or leveraged by other AI agents.
If that sounds appealing to you, you can fetch a .deb from:
r/linuxadmin • u/Arelvox • 4d ago
Java4Shell (CVE-2026-114938) is confirmed and patched. Raspberry Pi/ARM is safe, here's a 10-second check for x86 boxes
arelvox.comThe "unconfirmed OpenJDK RCE" from yesterday is now confirmed by OpenJDK and patched by Oracle. Short version for self-hosters:
What: heap overflow in Java's hardware-accelerated Base64 decoder. Calif showed pre-auth RCE with default configs on Keycloak, Jira, Confluence, Artifactory, Jenkins and PingFederate.
You need ALL of these to be exposed:
- Java 17.0.6+, 21, 25 or 27 (any vendor: Temurin, Corretto, Zulu...). Java 11 and normal Java 8 are fine.
- x86-64 CPU with AVX-512 VBMI (Intel Ice Lake Xeon+, AMD Zen 4+). ARM / Raspberry Pi is NOT affected.
- Untrusted Base64 reaching the decoder (Basic auth, JWTs, uploads...).
Check your CPU (all 4 flags = affected):
grep -m1 '^flags' /proc/cpuinfo | tr ' ' '\n' | grep -E '^(avx512f|avx512bw|avx512vbmi|bmi2)$'
Check Java inside containers, not just the host:
docker exec -it CONTAINER java -version
Fix: Oracle builds ending in .1 (17.0.21.1, 21.0.13.1, 25.0.5.1, 27.0.1.1); other vendors are rolling out. Until then, -XX:-UseBASE64Intrinsics stopped the overflow in every tested version (needs a restart/recreate). Don't use -XX:DisableIntrinsic=_base64_decodeBlock on its own, it stops the JVM from starting.
No exploitation in the wild reported yet, and VulnCheck says full RCE needs per-app work, but crashing a JVM is easy.
r/linuxadmin • u/CautiousCat3294 • 2d ago
Linux troubleshooting scenario: Application suddenly disappears without any error logs. Where would you start?
Imagine you're handling a production Linux server.
A Java application was running normally, but suddenly the process disappears.
You check the application logs and notice something strange:
- No Java exception.
- No stack trace.
- No graceful shutdown message.
- Logs simply stop at a particular timestamp.
- The application process is no longer running.
You check the application configuration, but nothing unusual stands out.
Now, here's the question:
Would you immediately investigate the application, or would you look at the Linux kernel first?
One possibility worth investigating is the Linux OOM (Out of Memory) Killer.
When the kernel cannot reclaim enough memory, it may terminate a process to protect the system from memory exhaustion.
I'd start by checking:
dmesg | grep -i oomAnd:
journalctl -k | grep -i oomBut confirming an OOM event is only the beginning.
I'd also want to understand:
- Which process was killed?
- Was it a system-wide OOM or a container memory limit?
- What caused the memory pressure?
- Was there a memory leak or sudden workload spike?
- What evidence would you collect before restarting the application?
I'm interested in how experienced Linux admins and production support engineers approach this situation.
r/linuxadmin • u/Ecno92 • 2d ago
RAM compression is not enabled by default. It is awesome.
I recently started to run out of memory. Where I would previously work on one thing at the time. With agentic engineering I suddenly work on 3-4 things in parallel. Each session spinning up a full version of the application that I am working on.
This means that the demand of RAM has suddenly spiked and processes were killed because of the lack of available memory on my system.
I enabled zram. Now I can fit much more data in my memory. I considered zswap as well, but I have no specific reason to pick one over the other.
My experience I wrote down in a simple article https://curiousbits.nl/blog/do-not-buy-ram-compress/
Other operating systems like Fedora, Windows and MacOS already enable this technique by default. In some cloud environments it is also applied (small amazon linux instances).
I believe that in the current times where RAM is expensive, we should consider enabling it by default for desktop users and we should experiment more to see how well it works in the cloud. (Where CPU capacity is often related to Memory capacity)
What is your view on this? Have you tried deploying this technique for larger applications already?
r/linuxadmin • u/buzzqw • 3d ago
I missed MobaXterm on Linux, so I built PCM: SSH, SFTP, RDP, VNC, Mosh and serial in one GTK3 app
Hi all,
At work I used MobaXterm for years, and when I moved to Linux full time I never found a single tool that did everything in one window. So I started writing my own, and after a lot of releases I think it's mature enough to share.
**PCM (Python Connection Manager)** is a connection manager for Linux and FreeBSD, written in Python with GTK3 and a native VTE terminal. It runs natively on Wayland and X11.
Highlights:
- **One window for everything**: SSH, SFTP, FTP/FTPS, RDP, VNC, Telnet, Mosh, Serial, SSH tunnels, plus plugins for SPICE, AWS SSM, kubectl exec and Docker
- **Dual-pane SFTP browser** with a transfer queue (progress, speed, ETA), drag & drop and local↔remote directory sync
- **Live monitoring side panel** for SSH sessions: CPU, RAM, processes (with kill), disks, network sparkline and streaming logs, all over the existing SSH channel
- **Passwords never on the command line**: PCM types them into the PTY when the server asks, so nothing shows up in `ps`
- **Encrypted credentials** with a master password, **KeePassXC integration** and **TOTP/2FA auto-type**
- **Session tree** with groups, tags, templates with inheritance, jump hosts, broadcast to multiple terminals
- **Importers** for Remmina, PuTTY, `~/.ssh/config`, Remote Desktop Manager and MobaXterm
- Interface in 5 languages
Install: AppImage or `.deb` from the releases page, or from source with `bash setup.sh` (it detects Debian/Ubuntu, Arch, Fedora, RHEL, openSUSE and FreeBSD). License EUPL-1.2.
Repo: https://github.com/buzzqw/Python_Connection_Manager
There's a comparison table with MobaXterm, Remmina, Ásbrú and mRemoteNG in the README. If you spot a mistake there, tell me and I'll fix it. Any feedback from people who manage a lot of servers is very welcome.
https://github.com/buzzqw/Python_Connection_Manager/blob/master/immagini/pcm66.png
https://github.com/buzzqw/Python_Connection_Manager/blob/master/immagini/pcm76.png
https://github.com/buzzqw/Python_Connection_Manager/blob/master/immagini/pcm87.png
*Disclosure: this project was developed with the support of AI assistants (Claude and DeepSeek).*
r/linuxadmin • u/rustyantenna • 4d ago
What do hiring teams mean by "python automation" in Linux deployment and configuration context
Hello,
Ive been looking at job postings and I see often they say python automation. The roles are infra, linux, systems engineer, etc.. What exactly do employers mean when they say python automation in this context?
From my experience for automating deployments on bare metal youd use things like poe, wal, pxe, for virtual/cloud terraform, for images packer, for configuration ansible and bash, etc.. Where exactly does python fit in?
Can you give some examples of workflows where python can be beneficial to automate (particularly in a linux related context).
Or do they mostly mean developing internal tooling that provides some custom functionality (or perfomance benefit vs bash) which the above tools lack of?
r/linuxadmin • u/charanreddy234 • 3d ago
Enabling DNSSEC on AD-integrated DNS
Hey everyone, we have an audit requirement to enable DNSSEC on our internal AD-integrated DNS hosted in domain controllers and I’ve never set this up before. Every guide I find only covers the Windows side (like configuring NRPT), and I have zero clue what needs to be done on Linux.
Could anyone help me know what exact config is needed on our Linux machines so they work properly with AD DNSSEC?
Here is our current setup:
• Linux clients are joined to the AD domain they use Winbind for authentication. And those are suse
They talk directly to our AD DNS servers over a ipsec tunnel backed by palo Alto (no separate resolvers in between).this is basic client and AD communication. And I wonder where this validation can happen since no resolver exists. Kindly help
r/linuxadmin • u/Pauliuss • 3d ago
Who is using Fastpanel.direct - check your Vps'es
reddit.comr/linuxadmin • u/jlrueda • 4d ago
Follow-up: I built the "Linux time travel" tool I asked about here. Looking for people to try it
About a month ago I asked here if a tool to see ps, ss or lsof output at a past moment was worth building:
https://www.reddit.com/r/linuxadmin/comments/1wyhwfe/is_a_linux_time_travel_tool_worth_building/
I got some fair pushback (sar, atop, below, Prometheus, zabbix) and a few people who were curious. I looked at all of them. They cover a lot, but none of them could tell me which process had which connections open, or what lsof showed for a process at 02:45 last night. So I built it, mostly to find out if it could be done without hurting the server.
I wanted to name it "command line time travel" but then a though about the consequences of such a name and It's called clifr instead (command line flight recorder) and has two parts:
- clifrd runs as a service and records processes, process starts and exits (through eBPF, so a process that lives 5 ms still shows up), TCP connections with the process that owns them, files held open, mounts, kernel messages and the usual system counters. It keeps 72 hours in /var/lib/clifr, 2 GB max.
clifr rebuilds the output of the normal commands for a past moment or a time range. It doesn't run the command, it rebuilds the output from the recording:
clifr --at "02:45" "ps aux"clifr --since "02:00" --until "03:00" "ss -tanp dst10.0.4.21and dport = :5432"clifr --since "1 hour ago" events # starts, exits, OOM kills, mountsclifr diff --at "14:00" --at "14:20" "ss -ltnp"
There are about 40 commands so far: ps, top, pidstat, lsof, ss, netstat, free, vmstat, iostat, mpstat, df, ip, dmesg, systemctl list-units, lsblk, and more and all are 100% accurate.
The first thing people asked last time was the cost, so here are the numbers:
┌───────────┬─────────────────────┬─────────────────────────┬─────────────────────────┐
│ │ Quiet server │ Mixed load (4-day soak) │ Busy server (benchmark) │
├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
│ CPU │ 0.35% of one core │ 0.5-0.6% of one core │ 1.9% of one core │
├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
│ RAM │ ~90 MiB │ ~80-120 MiB │ ~290 MiB (335 max) │
├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
│ Disk kept │ ~20 MiB/day │ ~100 MiB/day │ ~4 GiB/day │
├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
│ Network │ none, no ports open │ none │ none │
└───────────┴─────────────────────┴─────────────────────────┴─────────────────────────┘Disk use depends heavily on how busy the server is.
clifr is not a monitoring tool and has no alerts. It's not a security audit tool either: it doesn't record file reads or writes, and root can stop it. It's for the "what was going on at 02:45" question, after the fact.
Status: it's a prototype, version 0.1.0. It hasn't run on a real production server yet (a friend starts one this week), and all the load has been synthetic. AGPL-3.0, no telemetry, nothing leaves the box.
So if you have a test or dev box (RHEL 8/9, Rocky, Alma), I'd really appreciate it if you tried it and told me what breaks or what looks wrong. The most useful thing for me is comparing what clifr shows with what you know was really happening.
curl -LO https://github.com/jlruedab/clifr/releases/download/v0.1.0/clifrd-0.1.0-1.x86_64.rpm
curl -LO https://github.com/jlruedab/clifr/releases/download/v0.1.0/clifr-0.1.0-1.x86_64.rpm
sudo dnf install ./clifrd-0.1.0-1.x86_64.rpm ./clifr-0.1.0-1.x86_64.rpm
sudo systemctl enable --now clifrd
# wait a few minutes, then:
clifr --at "5 minutes ago" "ps aux"Repo, with an install guide, a user guide with use cases, and all the benchmark numbers:
https://github.com/jlruedab/clifr
And thanks to everyone who commented on the first post, including the harsh ones. It made this better.
r/linuxadmin • u/Due_Use722 • 3d ago