r/ROCm • • 7m ago

Inkling-2-Mini at the full 1M context on one RX 7900 XTX (Windows, ROCm): 9/9 retrieval, 37 min prefill, ~10 tok/s decode, and how it compares to llama.cpp

1 Upvotes

TL;DR: I optimised a native engine (based on Strata's strata-ink2) that runs Inkling-2-Mini (looped 26B MoE, 1M context) on a single 24 GB RX 7900 XTX with 32 GB RAM, on Windows. It is the real 1,048,576-token context, int8 KV cache, Unsloth UD-Q8_K_XL weights, nothing truncated or skipped. A 1M prompt takes ~37 min to read, then it decodes at ~10 tok/s. llama.cpp on the same box does ~1 tok/s with the experts on the CPU and can't do 1M at all.

Setup

  • RX 7900 XTX 24 GB, 32 GB RAM, Windows 11, ROCm/HIP (gfx1100) from AMD's TheRock wheels. No WSL.
  • Weights: Unsloth UD-Q8_K_XL, converted losslessly (the NVFP4 experts stay bit-exact). KV cache: int8. No layer or loop skipping, no eviction, no summarising.
  • About 15 GiB of experts sit in pinned host RAM and the hot ones are cached in VRAM. At 1M the KV cache alone is ~17 GB, so only ~9 % of the experts fit in VRAM.

Numbers (same machine, logs are in the repo)

llama.cppthis engine, first HIP portthis engine now
Prefill, 32K prompt452 tok/s1,677 tok/s~3,090 tok/s
Prefill, 256K promptnot measured272 s119 s
Prefill, 1M promptdoes not fit52 min37 min
Decode, 16K context~1 tok/s73 tok/s81 tok/s
Decode, 256K contextnot measured18 tok/s~52 tok/s
Decode, 1M contextdoes not fit~7 tok/s~10.5 tok/s

1M retrieval: 9/9 exact matches (8 planted facts at different depths plus a multi-hop question). Accuracy against a llama.cpp CPU reference: mean KL ~0.0023 over 4x512 tokens (llama.cpp's own HIP backend: 0.0030-0.0033).

What made it faster

  • A fused relative-bias flash attention for the int8 KV (WMMA). Rewriting it so that P stays in registers gave another ~1.5x on the attention kernel.
  • Split-key decode attention: reads the KV at ~530 GB/s instead of ~260.
  • fp16-output projection GEMMs (about 10 -> 90 TFLOP/s in isolation).
  • One skinny gate GEMM was running at 1.7 TFLOP/s; splitting it along K fixed that.
  • A guard for Windows silently moving a big buffer out of a full card into system RAM (it cost +31 ms per token once).
  • Save/load of the KV state to disk, because a 1M prefill should not happen twice.

What did not work: a register prefetch in the attention kernel (it spilled), speculative decoding with the model's own draft head (a verify costs ~3 decode steps and gives ~2.3 tokens, so it is slower), and smarter expert-cache policies (at a few percent cache even an oracle policy saves only ~40 % of the traffic).

Caveats, please read

  • One machine, and mostly one run per number. At 256K the decode is sometimes ~2x slower in a fresh process (about 1 start in 6) and I have not found out why. The table shows the fast state.
  • The llama.cpp column is a single configuration (llama-batched-bench, experts on the CPU, flash attention on). Its own expert cache could do better. At 1M it builds the relative attention bias as a dense tensor (~51 GB per layer), so it simply does not fit.
  • The 1M test is synthetic exact-match retrieval, not a general long-context quality benchmark.
  • The newest attention kernel is checked up to 256K; the 1M run used the previous one.
  • HIP only. Not built for CUDA or SYCL.
  • Upstream Strata itself was not measured on this machine.

Links


r/ROCm • • 11h ago

Hanuman

Post image
0 Upvotes

r/ROCm • • 20h ago

Hierarchos-Native v0.2.5 is finally out! P3 WMMA update

3 Upvotes

so, after quite a bit of work since the v0.2.0 release, i've finally managed to get v0.2.5 published!

for anyone who followed the previous release, v0.2.0 was where i first introduced the opt-in RDNA3 WMMA implementation. since then i've been working on improving the Full FP16 path, specifically its numerical accuracy and performance, without having to sacrifice the execution structure that was already working.

the biggest change in this release is something i've been calling P3.

previously, the Full FP16 WMMA kernel merged its eight accumulator-bank partials using multiple intermediate FP16 additions. this meant that the result was rounded to FP16 several times along the way. P3 changes that final merge so all eight partials are summed in FP32, with just one final rounding to FP16.

the actual two-wave workgroup structure, K-slice stream, bank assignment, staging, cooperative multiplication and barriers are all preserved. i wanted to improve the accuracy of the final result without having to redesign the kernel all over again.

some of the results from the new qualification:

  • 37/37 numerical gates passed across all eight qualification seeds.
  • 32/32 preserved K=96 outputs were bit-identical to the previously qualified P3 GPU results.
  • REF64F16 maximum error improved from 0.03035259 to 0.02593803.
  • 31 of 32 cases improved, with zero cases getting worse.
  • The maximum tile16 distance is 0.03125, meeting the existing hard numerical limit.

i also ran another full performance sweep. this time using 24 paired measurement blocks per workload, GPU timestamps, and corrected 95% confidence intervals.

the measured improvements against the preserved Main kernel baseline were:

  • 2048×768×2048 — 1.0243×
  • 512×2048×768 — 1.1246×
  • 512×768×768 — 1.1163×
  • 128×8192×768 — 1.2722×

all four protected workloads passed independently, including the lower bounds of their confidence intervals. the largest measured improvement was around 27%.

just to clarify, these are per-dispatch GPU GEMM measurements, not end-to-end training speedups. i don't want to imply that an entire training run is suddenly 27% faster when that isn't what was measured.

there were some other changes too.

i tightened up the production dispatch admission so the profitability predicate is checked against each actual batch, rather than relying only on the maximum batch size supplied when creating the projection. the existing 16,000,000 MAC profitability floor has not changed.

i also went through the actual portable binaries and verified that they dispatch the P3 kernel when an eligible workload is supplied. the packaged trainer produced 8 cooperative P3 dispatches, and the packaged logits executable produced another 3. disabling WMMA resulted in zero cooperative dispatches, as expected.

the numerical comparison for the packaged logits test covered 262,144 finite elements, with a maximum absolute difference of 0.000738330 and RMS difference of 0.000122237.

the new release is available as a portable Windows x64 package too.

i've included seven executables, and the package was built with a statically linked MSVC runtime. there's no need to install Rust, Cargo, Python, PyTorch, Visual Studio, or the MSVC redistributable just to run it.

you will still need a compatible Vulkan-capable GPU driver and the Vulkan runtime loader. model weights aren't included.

one thing i want to make clear is that WMMA is still opt-in globally. the default is still off, and you have to explicitly select full-fp16 to use the P3 path. P3 is simply the default implementation within that mode. the Mixed precision path is separate and hasn't been replaced.

the hardware qualification was performed on my AMD Radeon Graphics adapter, PCI ID 1002:15bf (RDNA3 / Phoenix). i haven't qualified this on every RDNA3 GPU, RDNA3.5, RDNA4, Intel or NVIDIA hardware, so please don't assume universal compatibility.

there are still things i haven't been able to test, including independent clean-machine/second-device acceptance, GUI interaction, Windows 10 acceptance, and code signing. the executables are currently unsigned.

anyway, i'm really happy to finally have this one out. v0.2.0 was the first public release of the WMMA implementation, and v0.2.5 is a pretty substantial improvement to that foundation. a lot of time went into checking that the numerical improvements were real, that the performance was actually measurable, and that the distributed binaries were doing what they were supposed to do.

Release: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.5

Source: https://github.com/necat101/Hierarchos-Native

thank you to everyone who's been following the project, testing things, reporting issues, or just keeping an eye on development. it genuinely means a lot! thank you so much everyone!! :3 also, sorry for the AI posts lately, I've been trying to save as much time as possible to work on the project itself due to live events getting tough! this one is completely human written! :3


r/ROCm • • 21h ago

Hy4 Flash added to int8-vllm: 4x MI100 C8 tg128 222.4 tok/s, pp512 1,655 tok/s

Thumbnail reddit.com
3 Upvotes

r/ROCm • • 1d ago

Ran

Post image
0 Upvotes

Ram ram


r/ROCm • • 1d ago

Trained a 1.18B code model from scratch on 8× MI325X: 151B tokens at ~35% MFU (numbers, lessons, and a tokenizer whitespace hack)

Post image
2 Upvotes

Quick progress report on CodeSymb-1B, a 1.18B-parameter code model trained from scratch on a Supermicro H14 8U with 8× AMD Instinct MI325X (JumpStart program). All numbers below come from the run's own logs (9,933 metric records, one every 20 steps).

Results

- 151.28B tokens trained from scratch (loss 5.33 → 1.065)

- 488–508K tokens/s median by phase, 34.6% median MFU (35.5% best), BF16

- 89.7 h of training inside a ~111 h window (81%)

- Stack: Ubuntu 24.04, ROCm/HIP 6.3, PyTorch 2.9.1+rocm6.3, DDP + torch.compile — PyTorch's built-in kernels, tuned with TunableOp

What mattered most on MI325X

  1. TunableOp is the single biggest lever. Without it: 300–408K tok/s. With tuned GEMMs: ~505K. First tuning took ~26 min; saving the per-GPU result files and reusing them cut startup to ~20 s. Save these files next to your checkpoints.
  2. 256 GB HBM3e changes the batch math. 48 sequences × 2,048 tokens per GPU fit with no gradient accumulation (786K tokens per step).
  3. The host CPUs can run your whole data pipeline beside training. On 2× EPYC 9575F we pretokenized 260B tokens (30.8 h), deduplicated 247M documents (14 min) and built a 22.5B-token SFT set (6.2 h) while training stayed at 488–496K tok/s.
  4. Make stops lossless. A stop file triggers a full save (weights, optimizer, data position); full checkpoints every 30 min, milestones every 10B tokens. Every restart was cheap, including a clean stop 54 min before the Nov 20 maintenance cutoff.

A data fix worth sharing: our SentencePiece tokenizer (NFKC, no byte fallback) silently deleted newlines and indentation — fatal for Python. Instead of retraining the tokenizer, we mapped 7 token IDs that never occurred in the data to newline/tab/space runs. Same vocabulary, same model; at 130.7B tokens we switched to 189.5B never-seen, deduplicated tokens with formatting kept.

Honest notes: median throughput after one mid-run restart was 2–4% lower than before it, and we haven't isolated why (GPUs were running near their ~1,000 W limit). The base model is not instruction-tuned or evaluated yet.

What's next: pretraining to 300B, then SFT (22.48B tokens, 4,096 context, reasoning + grounded RAG answers with citations), GRPO + DPO, and HumanEval/MBPP evals — about 5 days of compute once access resumes.

Full benchmark report (PDF):

CodeSymb_MI325X_Benchmark_Report.pdf

Model card and checkpoints:

NiftyOracleAI/CodeSymb-1B-MI325X

Thanks to Supermicro and the JumpStart team for the access, and to AMD for the MI325X stack. Happy to answer questions on the ROCm setup — ask away.


r/ROCm • • 2d ago

Looking for someone to run the first real training runs of a new framework on ROCm (ResNet-20, then OLMo-150M)

7 Upvotes

I maintain flodl, a deep learning framework in Rust on top of libtorch. It recently gained ROCm support: it builds against the ROCm libtorch, links, and detects AMD cards. It has never trained on one, because I don't own any AMD GPU.

If you have a supported card, Linux with Docker, and about an hour, the issue below has the exact commands:

- ResNet-20 on CIFAR-10, 200 epochs (14 minutes on an RTX 5060 Ti, for scale)

- OLMo-150M pretraining, one pass over 20M tokens (wants 8 GB of VRAM or more)

Supported: ROCm 7.0 and 7.1, gfx908 to gfx1201, so RDNA 2 to RDNA 4 including Strix APUs, and MI100 to MI350. Vega 10 and Vega 20 are not covered.

If you have two or more cards, there is a multi-GPU variant, which would be the first RCCL run of this framework anywhere.

It is likely that it will break somewhere, but this is the point. Report on the issue, or here, and I will investigate.

https://github.com/flodl-labs/flodl/issues/80

Update, 21 Nov: it works. Two people ran it within a day, on three cards, and flodl now trains on AMD.

CardArchROCmResNet-20 accuracyOLMo-150M
Strix Halo iGPUgfx115110.291.96%completes
Radeon AI PRO R9700gfx120110.291.78%completes
RX 7900 XTXgfx11007.0 and 10.291.59% / 91.79%fails, under investigation
RTX 5060 Ti (reference)91.46%

Something useful beyond this project, from the first report: with the same libtorch 2.10, ROCm 7.0 and 7.1 segfault on gfx1151 and run about 5x slower than 10.2 on gfx1201. The full version matrix (7.0 to 10.2, both cards) is in the issue.

Thanks to dhekimian and DhruvDh, whose reports are in the issue. Still looking for: other cards, anyone who can reproduce the OLMo failure on gfx1100, and anyone with two AMD cards for a first multi-GPU run.


r/ROCm • • 2d ago

R9700 + 7900XTX + 7900 XT or 2 x R9700?

Thumbnail reddit.com
0 Upvotes

r/ROCm • • 2d ago

NEED HELP for TTS model, i have RX 6600 8gb VRAM

1 Upvotes

what should i do? as far as i have researched i can't use QWEN 3 TTS or any good one. anyone please tell me what should i do?


r/ROCm • • 2d ago

Curation vs Compaction for agentic work on budget hardware

Thumbnail
reddit.com
1 Upvotes

r/ROCm • • 2d ago

Strix Halo 2x Nightlies build

4 Upvotes

so i was benchmarking my own inference code for strix halo https://github.com/Legendofzaday/halodiffuse against comfyui and found out that rocm nightlies v2 wheels are twice as fast as the stable wheels on sd3.5 large turbo this was the case:

halodiffuse: 4.6s
comfyui nightly v2 wheels: 5.6s
comfyui stable: 11.8s

cold start times have also improved a lot:

halodiffuse: 9.1s
comfyui nightly v2 wheels: 22.6s
comfyui stable wheels: 31.2s


r/ROCm • • 3d ago

PART 2: 2x RX 7900 XTX + Qwen3.8-Flash-Next — EXL3, hipfire, Avalon, the rdna3 fork's NaN found, and every config quality-tested (spoiler: above ~3 bits it's a plateau)

28 Upvotes

A week ago I posted engine speeds for Flash-Next on my 2x 7900 XTX box. I got several interesting responses for more things to try, and software moves so fast in this space that several updates also came out, so here are the updates.

(Thanks especially to u/nasone32 (EXL3 fork), u/zeno_g2 (hipfire and others), u/sergitto11 (the #879/ROCm lead), u/xyzzing (the optimization ledger), and everyone who posted their own numbers.)

Same box: Ryzen 7 5800X, 64 GB DDR4, 2x RX 7900 XTX at PCIe 4.0 x8, now on ROCm 10.1. The speed bench is unchanged: 112 / 3.5k / 34k-token prompts, 512 out, temp 1.0 / top-p 0.95 / top-k 20, thinking on, MTP on, medians.

Critical note: quality is always checked, not just speed

Every engine and quant below also ran my private coding-agent benchmark: real bugs mined from ~10 years of my own git history, with hidden tests (results public at lab.kylebaker.io). I compare configs problem by problem on the same problems, not leaderboard averages, and use a sign test on the problems where they disagree. Since late September that's ~8,000 scored local runs and ~2,100 agent-hours. this is super slow and painful; my computer is pretty much running agentic coding benchmarks 24/7 trying to test all the models and engines and occassionally harness expierments.

Also critical: I'm not benchmaxing.

My speeds use the model's recommended sampling (temp 1.0, thinking on), not greedy, which flatters speed. The table is a fixed-prompt bench; the wall-time comparisons come from real multi-turn agent runs on hard bugs.

What that shows:

  • Above ~2.5–3 bits, Flash-Next quants are a plateau on agentic coding.
    • GSQ-RCO IQ3_XXS (76 GB), IQ3_S, UD-IQ4_XS, UD-Q4_K_XL (111 GB) and EXL3 4.05 are all within noise of each other.
    • Two runs of the same config disagree on about as many problems as two different quants do.
    • The only quant with a measurable loss is IQ1_M.
    • I thought about running q8 on the cloud to verify this, but it would cost me somewhere between $30 and $120... not worth it to me, unfortunately, as it's purely a theoretical question. (A full run on my bench suite tends to require roughly 5M generated tokens.)
  • "Does Strata trade quality for speed?": I see no sign of it. Strata's IQ3_XXS is my strongest Flash-Next config, ahead of every llama.cpp Flash-Next config I've run.
    • The two quality-trading flags u/nasone32 mentioned (STRATA_ROUTE_PRIOR, STRATA_ROUTE_TAIL_SKIP) are off on gfx1100. ROUTE_PRIOR is opt-in, and the rdna3 fork explicitly sets TAIL_SKIP=0.
    • But TAIL_SKIP is on by default on CUDA and on RDNA4 (gfx1200/1201) builds, so check yours if you compare quality on those.
    • Note that the RDNA3 fork of strata DOES seem to trade some quality for speed in my testing! but strata's iq3xxs and iq3s have both performed within noise ("slightly better", but within noise) of UD q4 k xl flash next on llama.
  • On 97 problems both ran, Strata (GSQ IQ3_XXS) solved 82 and llama.cpp (UD-Q2_K_XL) 79, with 10 vs 7 problems solved only by one side: noise.
  • On the one file I ran on both engines (UD-IQ4_XS), Strata solved 6 of 6 and llama.cpp 5 of 6. Small, but pointing the same way.

Results

engine (2x XTX unless noted)quantdecode t/s (112 / 3.5k / 34k)prompt t/s (3.5k / 34k)quality vs upstream (paired)
Strata 0.2.13.1GSQ IQ3_XXS79 / 100 / 841,440 / 2,496baseline
Strata-rdna3 fork, NaN-free variantGSQ IQ3_XXS~95–125 (119–127 in agent runs)2,418 / 3,346probably 2–5 pts weaker on the hardest problems (still testing)
EXL3 fork, 4.05 bpwEXL326 / 31 / 29686 / 1,122tie (188 vs 193 of 222; fork 183)
EXL3 fork, 3.05 bpwEXL349 / 50 / 50876 / 1,34919 vs 21 of 26 (noise)
Avalon 1200934a7, 1 cardAvalon IQ3_KS87 / 90 / 901,051 / 2,385not run at scale
Avalon 1200934a7, 2 cards, LLAMA_STAGES=0 + splitAvalon IQ3_KS64 / 75 / 641,741 / 2,434not run at scale

Notes

  • rdna3 fork NaN (#879), found. The lead that ROCm's runtime caused it didn't hold on my box: same NaN rate on ROCm 7.2.4 and 10.1. Bisecting with a replay of the requests that NaN'd pinned it to the fork's refill settings (Willing-Cover2914's reinstall fixing it is consistent with that).
    • With the 10 refill settings removed: zero NaNs across ~300 real agent runs.
    • It matches the the same refill machinery as the fork's issues #3 and #14.
    • I'm narrowing it to the exact key(s) now and will send the fork a PR soon.
    • Claude's write-up: lab.kylebaker.io/blog/hunting-a-nan-in-strata-rdna3
  • The fork vs upstream on quality.
    • Over 222 paired problems, upstream solved 193 and the fork 183.
    • I re-ran the 30 problems where they disagreed, fresh on both engines in the same week: upstream 22, fork 16.
    • Combined, about a 2% chance that's luck. So the rdna3 fork looks a few points weaker on the hardest problems.
    • While it looks dramatically faster, a weaker model writes more tokens... net: it's still ~16% less wall time per run. More paired runs are queued to pin down the size of the gap. But, again, at this point it's within 98% certainty that it's a few points weaker than upstream strata.
  • EXL3 (u/nasone32).
    • While in theory the 4.05 bpw EXL3 flash next quant should be even better than a traditional q4 quant... in my bench, it within-noise-ties IQ3_XXS on quality. IT does write ~28% fewer tokens, which suggests it may be stronger, but on 64 GB it can't keep 4.05 fully resident, so it runs at ~2.3x the wall time
  • This one was particularly interesting to me, as I was wondering if this would 'break the plateau', in theory this should be the 'strongest' flash next quant I can realistically run on my machine. Alas--given that I didn't actually see any real-world improvement, I can't say the 2.3x slower runtime (on my box) is worth it.
  • Likewise, ran the 3~bpw EXL3, still significantly slower than strata (though much faster than the 4.05bpw). Still all within noise on strength/solves on my benchmark. - Tip for agent use: set --conversation-cache-mib. It defaults to 0, and with sub-agents alternating conversations the engine re-read the whole ~150k-token prompt on every switch: ~30 s each, versus ~2 s with a 16 GB cache. That fixed a multi-hour time-out.
  • hipfire (u/zeno_g2). On the dense 27B it's shockingly good:
    • on one XTX, about 2x llama.cpp's speed: decode 94–98 vs 45–63 t/s, prompt reads ~2.1k vs ~0.9k;
    • no quality loss I can detect: 15 vs 17 of 23 paired, in about half the wall time; 60 more problems are running now.
    • Flash-Next on hipfire didn't work out on 64 GB: its 2-bit build only runs on one card, at ~15 t/s. Sad.
  • Haven't tried radiance or caminus yet.
  • Avalon.
    • Current main fixed the prompt-ring regression and the 32k out-of-memory I reported.
    • Two cards with LLAMA_STAGES=0 + manual split is the setup that works for me.
    • I'm re-checking default pipeline stages on the latest main, since issue #55 reports it working well on the same hardware.

The rdna3 fork's kernel switches, patched onto upstream Strata. The rdna3 fork's speed has two parts: tensor parallel across both cards and a table of gfx1100 kernel switches, mostly for reading prompts. I turned on 15 of those switches on upstream Strata 0.2.13.1:

  • Bench prompt reads roughly doubled: 2,549 / 4,886 t/s at 3.5k / 34k tokens, vs 1,440 / 2,499 plain. Decode unchanged.
  • Real agent runs: no wall-time gain. Prompt reading is a small share of an agent run, and cached prefixes mean only the new tokens get read.
  • Quality: 18 vs 21 solved on the same problems. Slightly worse, but within noise.
  • Strata 0.2.14+ now ships its own prompt kernels by default, so this is mostly moot; I'm testing 0.2.16 now (speed plus a quality pairing).

of note, contradicting claude's earlier assumption:

Our own first bisect round shows that with all the fork's tuning switches off and tensor parallel kept, decode dropped 23%. That's roughly back to upstream strata speed. That means most of the rdna3's fork's speed comes from its tuning switches, not from tensor parallel.

7900 XTX vs 3090?

I think dual 7900xtx is comparable to dual 3090 right now, which is kind of wild. Can anyone sanity check me on this? I bought my xtx's at about half the price of what 3090s were going for on facebook marketplace a few months ago (about $700~ vs. the about $1300~ 3090s were going for)

  • Against the best public dual-3090 Flash-Next recipe, on its own terms (greedy, thinking off): 2x XTX with the rdna3 fork matches or beats it on code and file edits (up to +31%), is 2–9% behind on prose, and reads long prompts at the same speed.
  • Watch out for greedy numbers: greedy flatters code by ~1.6x on this setup. Converted to normal sampling, I estimate the dual 3090 at ~90 t/s code and ~105–112 prose, against the fork's 104–111 and 95–109. Roughly a tie, with the XTX ahead on code.

How fast this moved

Two months ago, before strata, Flash-Next on this box meant llama.cpp on one card with experts streaming from RAM: ~15–20 t/s decode. Upstream Strata does 80–100 t/s (about 5x, no quality cost); the fork 105–125 (6–8x, a few points weaker on the hardest problems). For reference, llama.cpp on both cards does 31–36 t/s on the same bench today.

At this point RAM is my bottleneck; I'd be more likely to spend $700 floating around on more ram than another gpu.

Still testing more things, always: Strata 0.2.16 (all-own kernels, speed plus a quality pairing), the fork PR, more fork-vs-upstream paired runs, and Ternary-Bonsai-27B (5.9 GB, 1.58-bit) as a reference. If you're running radiance or caminus on RDNA3, I'd love numbers.


edit:

Strata 0.2.16 passed its speed and sanity check. Both fresh starts were clean against a same-session 0.2.13.1 baseline:

  ┌─────────────────────────┬───────────────────────┬─────────────┐
  │  2 XTX, IQ3_XXS, 262k   │      prompt read      │   decode    │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 112-token prompts       │ +14% (282 vs 248 t/s) │ +4%         │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 3.5k                    │ +20% (1,724 vs 1,432) │ +8%         │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 34k                     │ +11% (2,777 vs 2,498) │ −3% (noise) │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 256 / 512-token prompts │ +22% / +32%           │ ±, noisy    │
  └─────────────────────────┴───────────────────────┴─────────────┘
  • Correctness: both known-answer checks were right on every start, with no garbage.
  • Memory: it fits about 1,000 more experts in VRAM than 0.2.13.1 (18.8 vs 17.3 GiB of expert cache).
  • Quality pairing: its 26-problem pairing (three items) started automatically and is running now, with its 5-minute check armed.

Avalon stuff is more complicated, I still consider that project a WIP for my machine but am happily contributing my findings upstream as I find them and have high hopes for it going forward.


edit 2: Avalon, and a summary table


  ┌─────────────────────────────────┬───────────────────────────────┬─────────────────────────┬─────────────────────────────────────────────┐
  │  engine (two XTX unless noted)  │ decode t/s (112 / 3.5k / 34k) │ prompt t/s (3.5k / 34k) │                   quality                   │
  ├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
  │ upstream Strata 0.2.13.1        │ 74–79 / 75–100 / 77–84        │ ~1,440 / ~2,500         │ the reference                               │
  ├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
  │ Strata 0.2.16                   │ 77 / 81 / 75                  │ 1,724 / 2,777           │ being checked now (first item 11/12)        │
  ├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
  │ Avalon, one card                │ 86 / 92 / 91                  │ 974 / 2,768             │ untested                                    │
  ├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
  │ Avalon, two cards, #55 settings │ 87 / 103 / 98                 │ 1,718 / 2,772           │ untested                                    │
  ├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
  │ rdna3 fork, REFILL_SDMA off     │ 120–128 / 105–111 / 98–107    │ ~2,530 / 3,100–3,470    │ probably 2–5 pts weaker (workaround config) │
  └─────────────────────────────────┴───────────────────────────────┴─────────────────────────┴─────────────────────────────────────────────┘

avalon is running better, still doing tests. Hopeful that Avalon would maintain the quality of strata upstream while getting some of the speed the rdna3 fork of strata brings.

Another concern: does the newest strata, which gains some speed, do so at the expense of quality? that will take, yet again, a lot of tests.

sometimes I wish this space would calm down, I never get to the end of testing, there's always more improvements. Hard to complain about it.


edit 3:

tldr, avalon shows no quality loss like the rdna3 fork of strata, and 0.2.16 also looks like it got a speed boost with no quality loss. hipfire, further testing to validate is also still holding at showing no quality loss.

(Very promising initial results! Not statistically significant--there is likely some variation, but it may only be detectable with far more testing or on a more granular set of tests. This just means that quality may be better or worse, but it isn't a huge move in either direction--if I had any given engine trying to fix a bug, I wouldn't notice it being any worse, I'd say, at this stage.)

Results completed yesterday and today:

┌───────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────┐
  │           test            │                                          result                                          │
  ├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
  │ Avalon quality (one card, │ singles 22/24, against Strata 21 (0.2.13.1) and 22 (0.2.16). Composites: 7/12 and 8/9    │
  │  its IQ3_KS file, 26      │ bugs fixed, against Strata averages of 6.2 and 6.9. No sign of a quality cost.           │
  │ problems)                 │                                                                                          │
  ├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
  │ Strata 0.2.16 quality     │ singles 22/24, matching 0.2.13.1. Composites 5/12 and 3/9 (one run spun up 6 sub-agents  │
  │ (same 26)                 │ and hit the cap). No quality cost seen.                                                  │
  ├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
  │ hipfire vs llama.cpp,     │ L3: 17 vs 18 of 33 paired (noise) at half the wall time (7.2 vs 14.7 h). L2: only 4      │
  │ Qwen3.8-27B, round 2      │ paired so far                                                                            │
  ├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
  │ Task-driver plugin, pilot │ 12 of 27 bugs fixed, against ~15.8 in earlier runs; all 3 composites hit the 4 h cap, at │
  │  (27B Q4, laptop)         │  ~2x the tokens. The model marked about half its tasks done while their bugs still       │
  │                           │ failed. The plain arm, the real comparison, was interrupted before it finished.          │
  └───────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘

r/ROCm • • 3d ago

Four engines, seven models, one Radeon AI PRO R9700: local LLM benchmarks

Thumbnail reddit.com
2 Upvotes

r/ROCm • • 3d ago

Just Purchased RX 7900 XTX 24 GB

Thumbnail reddit.com
8 Upvotes

Any suggestions related to models are welcome.


r/ROCm • • 3d ago

DL research on AI 395/495 in PyTorch?

3 Upvotes

I'm not sure if this is the right place to ask this, since it seems like a lot of people are trying to run local LLMs, but can't hurt to ask.

Has anyone had any experience doing DL research using PyTorch on one of these AI 395/495 dev boxes (any brand)? AMD puts out their own version and they're are a number of vendors with their own versions. I'm wondering how stable it is. These are dramatically cheaper than the DGX Sparks, but I don't want to pull out what little hair I have left on top of my head trying to get it to run.

I would develop in Linux or WSL (don't care which). But I think I read you can't even do training under windows (not that I would do any serious research in Windows) so that suggests the drivers are still pretty immature and I should probably just pony up the $2000-3000 extra for a spark.

Thanks. If there's a better place to ask, please let me know.


r/ROCm • • 3d ago

My AMD V620 32GB is running solid after the power mod you guys recommended! Really appreciate the help! 🙌

Post image
4 Upvotes

r/ROCm • • 3d ago

3 x AI 9700 build - 96gb VRAM

Thumbnail gallery
9 Upvotes

r/ROCm • • 3d ago

AMD ROCm 10.2 Released With Latest Optimizations, Expanded Ryzen Laptop Support

Thumbnail
phoronix.com
81 Upvotes

r/ROCm • • 4d ago

[Help Wanted] Testing GPU detection & TFLOPS benchmark paths (NVIDIA/AMD) for my from-scratch SLM trainer

0 Upvotes

Hi guys,

I’m a 15-year-old developer building SLM-CREATOR—a from-scratch, hardware-aware trainer for small, hyperspecialized text models (capped at 100M parameters) in Python and Rust. Instead of using high-level wrappers or fine-tuning existing models, I’m hand-coding the entire pipeline (detection, sizing, BPE tokenizer, and training loop) to learn how everything works under the hood.

Project Link: github.com
Tracking Issue (#1): github.com

Where I need help:
I’m developing this on a CPU-only machine right now, so all of my GPU detection and benchmarking code has only been tested against mock CLI outputs. Since r/llmdevs folks run all sorts of local setups—multi-GPU rigs, hybrid laptop GPUs, and ROCm environments—I’d hugely appreciate it if anyone with an NVIDIA or AMD GPU could run architecture.py and see if it breaks or misidentifies anything on real hardware.

What the script is doing under the hood:

  • Vendor CLI Parsing: Detects NVIDIA (nvidia-smi), modern AMD (amd-smi), or legacy AMD (rocm-smi/rocminfo) to extract architecture, total VRAM (MiB), device index, and GPU name.
  • PyTorch Index Mapping: Sets CUDA_DEVICE_ORDER=PCI_BUS_ID before importing torch and matches vendor tool names against torch.cuda.get_device_name() so multi-GPU systems don't target the wrong device.
  • Matmul TFLOPS Benchmark: Runs a 12-second thermal soak (to get past laptop boost clocks), then takes the median of 10 runs of 20 4096x4096 matmuls. It checks torch.cuda.is_bf16_supported(including_emulation=False) (or compute capability >= 8.0) so it benchmarks native bf16 and falls back to fp32 rather than getting fooled by slow software-emulated bf16.
  • Deterministic Sizing: Pipes VRAM, training hours, TFLOPS, and token counts into a Rust binary (sizer_core) to compute the max parameter count across memory, Chinchilla compute-time, and data limits.

How to test it (takes ~2 mins):

  1. git clone https://github.com
  2. Set up a venv and install requirements.txt (with your GPU's PyTorch build).
  3. Build the Rust sizer: cd sizer_core && cargo build --release
  4. Run python architecture.py from the root folder.
  5. Paste your GPU model, OS, and the terminal output (or any tracebacks) in Issue #1 or in the comments here.

I'd love any feedback on CLI parsing edge cases, ROCm quirks, or how the thermal soak behaves on your card!


r/ROCm • • 4d ago

How to Underclock or Undervolt AMD V620 32GB?

Post image
4 Upvotes

r/ROCm • • 4d ago

Used RX 6800 XT vs 6900 XT (XFX Merc) for local LLMs – is the ~$63 premium worth it?

Thumbnail reddit.com
0 Upvotes

r/ROCm • • 5d ago

Handing two models at once in Radiance

Thumbnail
reddit.com
3 Upvotes

r/ROCm • • 5d ago

Calling all RX7900(gfx1100) class card owners!!!

Thumbnail
reddit.com
12 Upvotes

r/ROCm • • 5d ago

Radeon AI PRO R9700 (gfx1201) is an overpriced Ferrari shell that doesn't even roll — Stop misleading creators with "32GB VRAM"

0 Upvotes

Radeon AI PRO R9700 (gfx1201) is an overpriced Ferrari shell that doesn't even roll — Stop misleading creators with "32GB VRAM"

I bought the Radeon AI PRO R9700 (32GB) specifically for my YouTube video production workflow, expecting a reliable workstation GPU for ComfyUI image and video generation. Instead, I have spent 6 full days and nights just trying to make it run without crashing—zero actual production work done.
Here is the reality of using the R9700 on Ubuntu right now:

  • Terrible Ubuntu compatibility: The card wouldn't even load properly on the latest Ubuntu release, forcing me to downgrade Ubuntu itself just to get basic detection working.
  • Endless environment hell: After tweaking Python environments, Docker containers, and following community-recommended ROCm versions, the system still crashes 1 to 2 times every single day with Failed to pin framebuffer: -ENOMEM and desktop session freezes.
  • Severe CPU fallback & memory leaks: Instead of utilizing the GPU at 100% like NVIDIA cards do, PyTorch/ROCm constantly leaks memory and dumps heavy workloads onto the CPU (spiking CPU usage to 400%) until the system locks up.
  • Good only for LLM and Audio, terrible for Image & Video: While text generation (LLM) and music generation run fast, image and video workflows are a complete disaster. Especially FP8 models, which produce severe ghosting/artifacts and cause immediate crashes.

Even worse, this issue remains completely unresolved. For real video production, it performs worse and is far less stable than an 8GB NVIDIA GPU. Sure, it can spit out a 1-megapixel clip once, but sustained, long-hour production work is impossible. How can AMD dare call this a "PRO" card for people making a living?
With NVIDIA, at least creators know it's expensive and save up for something that actually works 24/7. AMD should stop baiting buyers with an "attractive price" and "generous 32GB VRAM" when the software is only half-baked. Be honest and tell customers upfront that it's an incomplete product for specific workloads only—so fewer creators suffer the same loss and frustration I did.
Right now, owning this card feels like buying an expensive Ferrari shell that doesn't even roll—nothing more than an overpriced display piece. Who is going to compensate creators for all this lost time and stress?