I run a small open source tool, boxd, that sends builds from coding agents to spare Linux machines (rootless Podman containers or small virtual machines, one per git worktree). Three host-side mechanisms, and what I got wrong first.
Admission by memory
- All boxes live in one control group with
memory.max and memory.swap.max (defaults 85 and 25 percent of host memory), plus per-box limits. - A new machine is admitted only if measured use plus its size fits the cap, and the sizes of running machines plus its size fit the cap too. Measuring alone fails: a young machine has not claimed its memory yet, so twelve 4 GB machines once got in on a 28 GB cap.
- Otherwise: exit 75 and a message with the numbers, for example
no room for another virtual machine (12288 MB of running machines + 4096 MB used by container boxes + 12288 MB asked > 24576 MB, 100% of the 24576 MB cap); wait for one to end or stop one. With --wait the client asks every 30 seconds and callers are served in the order they first asked. Without it, the client tries the next host. - Lesson: swap on zram is RAM. My count missed swapped memory. A group read 23713 MB with 7.7 GB more in swap and kept admitting machines until the kernel killed an 8 GB one. Now
memory.swap.current is added, and half the swap the group may use on zram comes off the default cap. - Each machine records its real peak need, and removal says so:
it needed 2900 MB of its 8192 MB at most.
The work area and its cleanup
- It is its own btrfs subvolume, so system snapshots do not keep build data.
- Whole box folders go, never single files: unused for
BOXD_KEEP_DAYS (14), then the least recently used one per pass once the disk passes BOXD_EVICT_AT (70 percent) until it is under BOXD_EVICT_TO (60). The folder of a deleted worktree goes when the client next notices. - On a disk shared with the system, setup picks lower levels: current fill plus 20 points (at most 70), 10 below that, and 3 days.
- Lesson: never judge a cache file's age by modification time. Package tools keep the upstream's old times on unpacked files, so one release deleted live cargo, Maven and npm cache files.
boxd-reset swaps in a clean subvolume in seconds. It refuses while anyone has an ssh session or holds boxd-keepawake.
A host that sleeps
The optional power module (boxd-idle) suspends after 20 quiet minutes: no ssh connections, load under 0.5, no boxd-keepawake lock. Before sleeping it arms wake on network and, if the disk is at BOXD_RESET_AT (85) percent, runs boxd-reset. The client sends a wake packet before each ssh; the wait is about 15 seconds.
Honest notes
All code was written by Claude Code, an AI coding agent, with me directing and testing on real hosts. It has been public for a few days. Host packages exist for Void Linux and Debian 13 only. MIT license: https://github.com/gsalucci/boxd
How do you handle memory admission on a host with zram swap, and what cleanup budget do you give build data on a disk shared with the system?