edit / update (nov 18):
got a lot of good advice here so i tried some of it, and a few things didn't go how i expected.
i thought tagging tests by area would be enough to pick what to run, but when i checked it on one change the tags missed 47 of the 91 test files that a list based on what imports what caught, including one that actually mattered. so i dropped the tags for now and i'll try them again after the cleanup. right now it's just the change's own tests while i'm writing, then everything that depends on the changed files when i push, and the full suite still runs before anything lands.
i figured the quick set of tests would take about a minute. turns out it was more like 7 and a half and didn't catch anything extra, so that's gone too.
i also tried skipping the local check before a train, thinking it'd speed things up. it didn't really, 133 min vs the usual 112-141, so i put it back.
and i was off on runner time, it's more like 5-7 hours depending on the train, the last one was about 5.4.
the bigger change is i'm reorganizing the code into clearer layers so a change doesn't touch as many places. i'm doing that before this release now instead of after, so the release will be later than i said. hoping to have it in before it ships if it passes everything, no results yet though. making the timing tests wait for a quiet laptop on their own, a cpu time check on one of them, and maybe trying testmon are all on my list for after that. not changing the timing limits.
questions for people who've done this on a big python desktop app:
- how did you split yours into layers so each part only depends on the ones below it, and how many layers or folders ended up being enough?
- how do you keep it from tangling back up over time?
- i'm thinking layers on the outside with feature folders inside the parts that change most. has anyone done it that way, and did it hold up?
solo dev building a windows desktop app, most of the coding is done by claude code sessions (earlier post about that). the bottleneck now is testing. the sessions spend more time waiting on CI than writing code, so i'm looking for ideas
The setup:
- the app drives blender, unity 2022.3 and the vrchat sdk, so the real tests need windows
- 4 self-hosted github actions runners, all on my one laptop (ryzen 9 7945hx 16 cores, rtx 4090 laptop 16gb, 64gb ram, 8tb nvme), one job per runner. the real blender/unity jobs can only use 3 of them, and up to 2 local test runs share the same laptop too
- self-hosted because github's hosted windows minutes cost double and i'm on a tight budget, so the laptop is the free option
- changes get batched into merge trains, 1-8 changes each, about 3 on average
What a train looks like:
- pipeline first: 16 real tool jobs (mostly blender, plus unity and a vrm validator) and fake-tool pytest in 4 shards, about 45-80 min
- then the full windows UI suite, ~25,600 tests in 3 shards, about 65 min. these run back to back on purpose because the timing tests flake when they share the machine with blender jobs, so that's about an hour of every train just waiting
- a train takes about 2 hours start to finish (83-145 min over the last 8)
- 7 of the last 8 landed, 4 clean first try. fix and retest loops took anywhere from 30 min to 2 hours
- laptop averages 55-60% cpu while all this runs and peaks at 100%, ram peaks around 80%
How much testing:
- every fix starts with a failing test, a review session checks it, the author runs the affected tests locally, then the train runs everything again in CI plus a local gate
- so the plain python tests end up running about 3 times. the real blender/unity tests did catch the only real failure in those 8 trains though
What i've tried:
- dev drive, only ~4% faster on my test so i didn't keep it
- sharding by recorded test duration, and splitting local runs by run time so they stop timing out
- allowing one re-run of only the failed jobs for flaky infra stuff
- tags so only affected tests run, built but not measured yet
Why one train at a time: changes often touch the same files, and the timing tests got flaky when all 4 runners were busy. so i'm not sure more runners even helps or if the real answer is a second machine
What i'm asking:
- am i over testing? how do you cut duplicate runs without losing coverage
- how do you pick tests by changed area without missing stuff that imports across areas
- is there a safe way to run 2 trains when changes overlap files, or is one at a time just right
- is a second machine or a cloud windows runner worth it for a ~2 hour cycle, or is there a better service for this
- how do you keep timing sensitive tests reliable when CI jobs share one pc
- could the plain python and fake-tool tests run on linux, or does that risk missing windows-only bugs
at the current pace my next release is about 3 weeks out and the one after that is a couple months, so anything that shortens the 2 hour loop helps a lot. thanks!