A week ago I posted engine speeds for Flash-Next on my 2x 7900 XTX box. I got several interesting responses for more things to try, and software moves so fast in this space that several updates also came out, so here are the updates.
(Thanks especially to u/nasone32 (EXL3 fork), u/zeno_g2 (hipfire and others), u/sergitto11 (the #879/ROCm lead), u/xyzzing (the optimization ledger), and everyone who posted their own numbers.)
Same box: Ryzen 7 5800X, 64 GB DDR4, 2x RX 7900 XTX at PCIe 4.0 x8, now on ROCm 10.1. The speed bench is unchanged: 112 / 3.5k / 34k-token prompts, 512 out, temp 1.0 / top-p 0.95 / top-k 20, thinking on, MTP on, medians.
Critical note: quality is always checked, not just speed
Every engine and quant below also ran my private coding-agent benchmark: real bugs mined from ~10 years of my own git history, with hidden tests (results public at lab.kylebaker.io). I compare configs problem by problem on the same problems, not leaderboard averages, and use a sign test on the problems where they disagree. Since late September that's ~8,000 scored local runs and ~2,100 agent-hours. this is super slow and painful; my computer is pretty much running agentic coding benchmarks 24/7 trying to test all the models and engines and occassionally harness expierments.
Also critical: I'm not benchmaxing.
My speeds use the model's recommended sampling (temp 1.0, thinking on), not greedy, which flatters speed. The table is a fixed-prompt bench; the wall-time comparisons come from real multi-turn agent runs on hard bugs.
What that shows:
- Above ~2.5–3 bits, Flash-Next quants are a plateau on agentic coding.
- GSQ-RCO IQ3_XXS (76 GB), IQ3_S, UD-IQ4_XS, UD-Q4_K_XL (111 GB) and EXL3 4.05 are all within noise of each other.
- Two runs of the same config disagree on about as many problems as two different quants do.
- The only quant with a measurable loss is IQ1_M.
- I thought about running q8 on the cloud to verify this, but it would cost me somewhere between $30 and $120... not worth it to me, unfortunately, as it's purely a theoretical question. (A full run on my bench suite tends to require roughly 5M generated tokens.)
- "Does Strata trade quality for speed?": I see no sign of it. Strata's IQ3_XXS is my strongest Flash-Next config, ahead of every llama.cpp Flash-Next config I've run.
- The two quality-trading flags u/nasone32 mentioned (
STRATA_ROUTE_PRIOR, STRATA_ROUTE_TAIL_SKIP) are off on gfx1100. ROUTE_PRIOR is opt-in, and the rdna3 fork explicitly sets TAIL_SKIP=0.
- But TAIL_SKIP is on by default on CUDA and on RDNA4 (gfx1200/1201) builds, so check yours if you compare quality on those.
- Note that the RDNA3 fork of strata DOES seem to trade some quality for speed in my testing! but strata's iq3xxs and iq3s have both performed within noise ("slightly better", but within noise) of UD q4 k xl flash next on llama.
- On 97 problems both ran, Strata (GSQ IQ3_XXS) solved 82 and llama.cpp (UD-Q2_K_XL) 79, with 10 vs 7 problems solved only by one side: noise.
- On the one file I ran on both engines (UD-IQ4_XS), Strata solved 6 of 6 and llama.cpp 5 of 6. Small, but pointing the same way.
Results
| engine (2x XTX unless noted) | quant | decode t/s (112 / 3.5k / 34k) | prompt t/s (3.5k / 34k) | quality vs upstream (paired) |
| Strata 0.2.13.1 | GSQ IQ3_XXS | 79 / 100 / 84 | 1,440 / 2,496 | baseline |
| Strata-rdna3 fork, NaN-free variant | GSQ IQ3_XXS | ~95–125 (119–127 in agent runs) | 2,418 / 3,346 | probably 2–5 pts weaker on the hardest problems (still testing) |
| EXL3 fork, 4.05 bpw | EXL3 | 26 / 31 / 29 | 686 / 1,122 | tie (188 vs 193 of 222; fork 183) |
| EXL3 fork, 3.05 bpw | EXL3 | 49 / 50 / 50 | 876 / 1,349 | 19 vs 21 of 26 (noise) |
| Avalon 1200934a7, 1 card | Avalon IQ3_KS | 87 / 90 / 90 | 1,051 / 2,385 | not run at scale |
Avalon 1200934a7, 2 cards, LLAMA_STAGES=0 + split | Avalon IQ3_KS | 64 / 75 / 64 | 1,741 / 2,434 | not run at scale |
Notes
- rdna3 fork NaN (#879), found. The lead that ROCm's runtime caused it didn't hold on my box: same NaN rate on ROCm 7.2.4 and 10.1. Bisecting with a replay of the requests that NaN'd pinned it to the fork's refill settings (Willing-Cover2914's reinstall fixing it is consistent with that).
- With the 10 refill settings removed: zero NaNs across ~300 real agent runs.
- It matches the the same refill machinery as the fork's issues #3 and #14.
- I'm narrowing it to the exact key(s) now and will send the fork a PR soon.
- Claude's write-up: lab.kylebaker.io/blog/hunting-a-nan-in-strata-rdna3
- The fork vs upstream on quality.
- Over 222 paired problems, upstream solved 193 and the fork 183.
- I re-ran the 30 problems where they disagreed, fresh on both engines in the same week: upstream 22, fork 16.
- Combined, about a 2% chance that's luck. So the rdna3 fork looks a few points weaker on the hardest problems.
- While it looks dramatically faster, a weaker model writes more tokens... net: it's still ~16% less wall time per run. More paired runs are queued to pin down the size of the gap. But, again, at this point it's within 98% certainty that it's a few points weaker than upstream strata.
- EXL3 (u/nasone32).
- While in theory the 4.05 bpw EXL3 flash next quant should be even better than a traditional q4 quant... in my bench, it within-noise-ties IQ3_XXS on quality. IT does write ~28% fewer tokens, which suggests it may be stronger, but on 64 GB it can't keep 4.05 fully resident, so it runs at ~2.3x the wall time
- This one was particularly interesting to me, as I was wondering if this would 'break the plateau', in theory this should be the 'strongest' flash next quant I can realistically run on my machine. Alas--given that I didn't actually see any real-world improvement, I can't say the 2.3x slower runtime (on my box) is worth it.
- Likewise, ran the 3~bpw EXL3, still significantly slower than strata (though much faster than the 4.05bpw). Still all within noise on strength/solves on my benchmark.
- Tip for agent use: set
--conversation-cache-mib. It defaults to 0, and with sub-agents alternating conversations the engine re-read the whole ~150k-token prompt on every switch: ~30 s each, versus ~2 s with a 16 GB cache. That fixed a multi-hour time-out.
- hipfire (u/zeno_g2). On the dense 27B it's shockingly good:
- on one XTX, about 2x llama.cpp's speed: decode 94–98 vs 45–63 t/s, prompt reads ~2.1k vs ~0.9k;
- no quality loss I can detect: 15 vs 17 of 23 paired, in about half the wall time; 60 more problems are running now.
- Flash-Next on hipfire didn't work out on 64 GB: its 2-bit build only runs on one card, at ~15 t/s. Sad.
- Haven't tried radiance or caminus yet.
- Avalon.
- Current main fixed the prompt-ring regression and the 32k out-of-memory I reported.
- Two cards with
LLAMA_STAGES=0 + manual split is the setup that works for me.
- I'm re-checking default pipeline stages on the latest main, since issue #55 reports it working well on the same hardware.
The rdna3 fork's kernel switches, patched onto upstream Strata. The rdna3 fork's speed has two parts: tensor parallel across both cards and a table of gfx1100 kernel switches, mostly for reading prompts. I turned on 15 of those switches on upstream Strata 0.2.13.1:
- Bench prompt reads roughly doubled: 2,549 / 4,886 t/s at 3.5k / 34k tokens, vs 1,440 / 2,499 plain. Decode unchanged.
- Real agent runs: no wall-time gain. Prompt reading is a small share of an agent run, and cached prefixes mean only the new tokens get read.
- Quality: 18 vs 21 solved on the same problems. Slightly worse, but within noise.
- Strata 0.2.14+ now ships its own prompt kernels by default, so this is mostly moot; I'm testing 0.2.16 now (speed plus a quality pairing).
of note, contradicting claude's earlier assumption:
Our own first bisect round shows that with all the fork's tuning switches off and tensor parallel kept, decode dropped 23%. That's roughly back to upstream strata speed. That means most of the rdna3's fork's speed comes from its tuning switches, not from tensor parallel.
7900 XTX vs 3090?
I think dual 7900xtx is comparable to dual 3090 right now, which is kind of wild. Can anyone sanity check me on this? I bought my xtx's at about half the price of what 3090s were going for on facebook marketplace a few months ago (about $700~ vs. the about $1300~ 3090s were going for)
- Against the best public dual-3090 Flash-Next recipe, on its own terms (greedy, thinking off): 2x XTX with the rdna3 fork matches or beats it on code and file edits (up to +31%), is 2–9% behind on prose, and reads long prompts at the same speed.
- Watch out for greedy numbers: greedy flatters code by ~1.6x on this setup. Converted to normal sampling, I estimate the dual 3090 at ~90 t/s code and ~105–112 prose, against the fork's 104–111 and 95–109. Roughly a tie, with the XTX ahead on code.
How fast this moved
Two months ago, before strata, Flash-Next on this box meant llama.cpp on one card with experts streaming from RAM: ~15–20 t/s decode. Upstream Strata does 80–100 t/s (about 5x, no quality cost); the fork 105–125 (6–8x, a few points weaker on the hardest problems). For reference, llama.cpp on both cards does 31–36 t/s on the same bench today.
At this point RAM is my bottleneck; I'd be more likely to spend $700 floating around on more ram than another gpu.
Still testing more things, always: Strata 0.2.16 (all-own kernels, speed plus a quality pairing), the fork PR, more fork-vs-upstream paired runs, and Ternary-Bonsai-27B (5.9 GB, 1.58-bit) as a reference. If you're running radiance or caminus on RDNA3, I'd love numbers.
edit:
Strata 0.2.16 passed its speed and sanity check. Both fresh starts were clean against a same-session 0.2.13.1 baseline:
┌─────────────────────────┬───────────────────────┬─────────────┐
│ 2 XTX, IQ3_XXS, 262k │ prompt read │ decode │
├─────────────────────────┼───────────────────────┼─────────────┤
│ 112-token prompts │ +14% (282 vs 248 t/s) │ +4% │
├─────────────────────────┼───────────────────────┼─────────────┤
│ 3.5k │ +20% (1,724 vs 1,432) │ +8% │
├─────────────────────────┼───────────────────────┼─────────────┤
│ 34k │ +11% (2,777 vs 2,498) │ −3% (noise) │
├─────────────────────────┼───────────────────────┼─────────────┤
│ 256 / 512-token prompts │ +22% / +32% │ ±, noisy │
└─────────────────────────┴───────────────────────┴─────────────┘
- Correctness: both known-answer checks were right on every start, with no garbage.
- Memory: it fits about 1,000 more experts in VRAM than 0.2.13.1 (18.8 vs 17.3 GiB of expert cache).
- Quality pairing: its 26-problem pairing (three items) started automatically and is running now, with its 5-minute check armed.
Avalon stuff is more complicated, I still consider that project a WIP for my machine but am happily contributing my findings upstream as I find them and have high hopes for it going forward.
edit 2: Avalon, and a summary table
┌─────────────────────────────────┬───────────────────────────────┬─────────────────────────┬─────────────────────────────────────────────┐
│ engine (two XTX unless noted) │ decode t/s (112 / 3.5k / 34k) │ prompt t/s (3.5k / 34k) │ quality │
├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
│ upstream Strata 0.2.13.1 │ 74–79 / 75–100 / 77–84 │ ~1,440 / ~2,500 │ the reference │
├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
│ Strata 0.2.16 │ 77 / 81 / 75 │ 1,724 / 2,777 │ being checked now (first item 11/12) │
├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
│ Avalon, one card │ 86 / 92 / 91 │ 974 / 2,768 │ untested │
├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
│ Avalon, two cards, #55 settings │ 87 / 103 / 98 │ 1,718 / 2,772 │ untested │
├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
│ rdna3 fork, REFILL_SDMA off │ 120–128 / 105–111 / 98–107 │ ~2,530 / 3,100–3,470 │ probably 2–5 pts weaker (workaround config) │
└─────────────────────────────────┴───────────────────────────────┴─────────────────────────┴─────────────────────────────────────────────┘
avalon is running better, still doing tests. Hopeful that Avalon would maintain the quality of strata upstream while getting some of the speed the rdna3 fork of strata brings.
Another concern: does the newest strata, which gains some speed, do so at the expense of quality? that will take, yet again, a lot of tests.
sometimes I wish this space would calm down, I never get to the end of testing, there's always more improvements. Hard to complain about it.
edit 3:
tldr, avalon shows no quality loss like the rdna3 fork of strata, and 0.2.16 also looks like it got a speed boost with no quality loss. hipfire, further testing to validate is also still holding at showing no quality loss.
(Very promising initial results! Not statistically significant--there is likely some variation, but it may only be detectable with far more testing or on a more granular set of tests. This just means that quality may be better or worse, but it isn't a huge move in either direction--if I had any given engine trying to fix a bug, I wouldn't notice it being any worse, I'd say, at this stage.)
Results completed yesterday and today:
┌───────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────┐
│ test │ result │
├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
│ Avalon quality (one card, │ singles 22/24, against Strata 21 (0.2.13.1) and 22 (0.2.16). Composites: 7/12 and 8/9 │
│ its IQ3_KS file, 26 │ bugs fixed, against Strata averages of 6.2 and 6.9. No sign of a quality cost. │
│ problems) │ │
├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
│ Strata 0.2.16 quality │ singles 22/24, matching 0.2.13.1. Composites 5/12 and 3/9 (one run spun up 6 sub-agents │
│ (same 26) │ and hit the cap). No quality cost seen. │
├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
│ hipfire vs llama.cpp, │ L3: 17 vs 18 of 33 paired (noise) at half the wall time (7.2 vs 14.7 h). L2: only 4 │
│ Qwen3.8-27B, round 2 │ paired so far │
├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
│ Task-driver plugin, pilot │ 12 of 27 bugs fixed, against ~15.8 in earlier runs; all 3 composites hit the 4 h cap, at │
│ (27B Q4, laptop) │ ~2x the tokens. The model marked about half its tasks done while their bugs still │
│ │ failed. The plain arm, the real comparison, was interrupted before it finished. │
└───────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘