r/ROCm • • 8h ago

Hierarchos-Native v0.2.5 is finally out! P3 WMMA update

3 Upvotes

so, after quite a bit of work since the v0.2.0 release, i've finally managed to get v0.2.5 published!

for anyone who followed the previous release, v0.2.0 was where i first introduced the opt-in RDNA3 WMMA implementation. since then i've been working on improving the Full FP16 path, specifically its numerical accuracy and performance, without having to sacrifice the execution structure that was already working.

the biggest change in this release is something i've been calling P3.

previously, the Full FP16 WMMA kernel merged its eight accumulator-bank partials using multiple intermediate FP16 additions. this meant that the result was rounded to FP16 several times along the way. P3 changes that final merge so all eight partials are summed in FP32, with just one final rounding to FP16.

the actual two-wave workgroup structure, K-slice stream, bank assignment, staging, cooperative multiplication and barriers are all preserved. i wanted to improve the accuracy of the final result without having to redesign the kernel all over again.

some of the results from the new qualification:

  • 37/37 numerical gates passed across all eight qualification seeds.
  • 32/32 preserved K=96 outputs were bit-identical to the previously qualified P3 GPU results.
  • REF64F16 maximum error improved from 0.03035259 to 0.02593803.
  • 31 of 32 cases improved, with zero cases getting worse.
  • The maximum tile16 distance is 0.03125, meeting the existing hard numerical limit.

i also ran another full performance sweep. this time using 24 paired measurement blocks per workload, GPU timestamps, and corrected 95% confidence intervals.

the measured improvements against the preserved Main kernel baseline were:

  • 2048×768×2048 — 1.0243×
  • 512×2048×768 — 1.1246×
  • 512×768×768 — 1.1163×
  • 128×8192×768 — 1.2722×

all four protected workloads passed independently, including the lower bounds of their confidence intervals. the largest measured improvement was around 27%.

just to clarify, these are per-dispatch GPU GEMM measurements, not end-to-end training speedups. i don't want to imply that an entire training run is suddenly 27% faster when that isn't what was measured.

there were some other changes too.

i tightened up the production dispatch admission so the profitability predicate is checked against each actual batch, rather than relying only on the maximum batch size supplied when creating the projection. the existing 16,000,000 MAC profitability floor has not changed.

i also went through the actual portable binaries and verified that they dispatch the P3 kernel when an eligible workload is supplied. the packaged trainer produced 8 cooperative P3 dispatches, and the packaged logits executable produced another 3. disabling WMMA resulted in zero cooperative dispatches, as expected.

the numerical comparison for the packaged logits test covered 262,144 finite elements, with a maximum absolute difference of 0.000738330 and RMS difference of 0.000122237.

the new release is available as a portable Windows x64 package too.

i've included seven executables, and the package was built with a statically linked MSVC runtime. there's no need to install Rust, Cargo, Python, PyTorch, Visual Studio, or the MSVC redistributable just to run it.

you will still need a compatible Vulkan-capable GPU driver and the Vulkan runtime loader. model weights aren't included.

one thing i want to make clear is that WMMA is still opt-in globally. the default is still off, and you have to explicitly select full-fp16 to use the P3 path. P3 is simply the default implementation within that mode. the Mixed precision path is separate and hasn't been replaced.

the hardware qualification was performed on my AMD Radeon Graphics adapter, PCI ID 1002:15bf (RDNA3 / Phoenix). i haven't qualified this on every RDNA3 GPU, RDNA3.5, RDNA4, Intel or NVIDIA hardware, so please don't assume universal compatibility.

there are still things i haven't been able to test, including independent clean-machine/second-device acceptance, GUI interaction, Windows 10 acceptance, and code signing. the executables are currently unsigned.

anyway, i'm really happy to finally have this one out. v0.2.0 was the first public release of the WMMA implementation, and v0.2.5 is a pretty substantial improvement to that foundation. a lot of time went into checking that the numerical improvements were real, that the performance was actually measurable, and that the distributed binaries were doing what they were supposed to do.

Release: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.5

Source: https://github.com/necat101/Hierarchos-Native

thank you to everyone who's been following the project, testing things, reporting issues, or just keeping an eye on development. it genuinely means a lot! thank you so much everyone!! :3 also, sorry for the AI posts lately, I've been trying to save as much time as possible to work on the project itself due to live events getting tough! this one is completely human written! :3


r/ROCm • • 10h ago

Hy4 Flash added to int8-vllm: 4x MI100 C8 tg128 222.4 tok/s, pp512 1,655 tok/s

Thumbnail reddit.com
3 Upvotes

r/ROCm • • 20h ago

Trained a 1.18B code model from scratch on 8× MI325X: 151B tokens at ~35% MFU (numbers, lessons, and a tokenizer whitespace hack)

Post image
2 Upvotes

Quick progress report on CodeSymb-1B, a 1.18B-parameter code model trained from scratch on a Supermicro H14 8U with 8× AMD Instinct MI325X (JumpStart program). All numbers below come from the run's own logs (9,933 metric records, one every 20 steps).

Results

- 151.28B tokens trained from scratch (loss 5.33 → 1.065)

- 488–508K tokens/s median by phase, 34.6% median MFU (35.5% best), BF16

- 89.7 h of training inside a ~111 h window (81%)

- Stack: Ubuntu 24.04, ROCm/HIP 6.3, PyTorch 2.9.1+rocm6.3, DDP + torch.compile — PyTorch's built-in kernels, tuned with TunableOp

What mattered most on MI325X

  1. TunableOp is the single biggest lever. Without it: 300–408K tok/s. With tuned GEMMs: ~505K. First tuning took ~26 min; saving the per-GPU result files and reusing them cut startup to ~20 s. Save these files next to your checkpoints.
  2. 256 GB HBM3e changes the batch math. 48 sequences × 2,048 tokens per GPU fit with no gradient accumulation (786K tokens per step).
  3. The host CPUs can run your whole data pipeline beside training. On 2× EPYC 9575F we pretokenized 260B tokens (30.8 h), deduplicated 247M documents (14 min) and built a 22.5B-token SFT set (6.2 h) while training stayed at 488–496K tok/s.
  4. Make stops lossless. A stop file triggers a full save (weights, optimizer, data position); full checkpoints every 30 min, milestones every 10B tokens. Every restart was cheap, including a clean stop 54 min before the Nov 20 maintenance cutoff.

A data fix worth sharing: our SentencePiece tokenizer (NFKC, no byte fallback) silently deleted newlines and indentation — fatal for Python. Instead of retraining the tokenizer, we mapped 7 token IDs that never occurred in the data to newline/tab/space runs. Same vocabulary, same model; at 130.7B tokens we switched to 189.5B never-seen, deduplicated tokens with formatting kept.

Honest notes: median throughput after one mid-run restart was 2–4% lower than before it, and we haven't isolated why (GPUs were running near their ~1,000 W limit). The base model is not instruction-tuned or evaluated yet.

What's next: pretraining to 300B, then SFT (22.48B tokens, 4,096 context, reasoning + grounded RAG answers with citations), GRPO + DPO, and HumanEval/MBPP evals — about 5 days of compute once access resumes.

Full benchmark report (PDF):

CodeSymb_MI325X_Benchmark_Report.pdf

Model card and checkpoints:

NiftyOracleAI/CodeSymb-1B-MI325X

Thanks to Supermicro and the JumpStart team for the access, and to AMD for the MI325X stack. Happy to answer questions on the ROCm setup — ask away.


r/ROCm • • 19h ago

Ran

Post image
0 Upvotes

Ram ram


r/ROCm • • 1d ago

Looking for someone to run the first real training runs of a new framework on ROCm (ResNet-20, then OLMo-150M)

7 Upvotes

I maintain flodl, a deep learning framework in Rust on top of libtorch. It recently gained ROCm support: it builds against the ROCm libtorch, links, and detects AMD cards. It has never trained on one, because I don't own any AMD GPU.

If you have a supported card, Linux with Docker, and about an hour, the issue below has the exact commands:

- ResNet-20 on CIFAR-10, 200 epochs (14 minutes on an RTX 5060 Ti, for scale)

- OLMo-150M pretraining, one pass over 20M tokens (wants 8 GB of VRAM or more)

Supported: ROCm 7.0 and 7.1, gfx908 to gfx1201, so RDNA 2 to RDNA 4 including Strix APUs, and MI100 to MI350. Vega 10 and Vega 20 are not covered.

If you have two or more cards, there is a multi-GPU variant, which would be the first RCCL run of this framework anywhere.

It is likely that it will break somewhere, but this is the point. Report on the issue, or here, and I will investigate.

https://github.com/flodl-labs/flodl/issues/80

Update, 21 Nov: it works. Two people ran it within a day, on three cards, and flodl now trains on AMD.

CardArchROCmResNet-20 accuracyOLMo-150M
Strix Halo iGPUgfx115110.291.96%completes
Radeon AI PRO R9700gfx120110.291.78%completes
RX 7900 XTXgfx11007.0 and 10.291.59% / 91.79%fails, under investigation
RTX 5060 Ti (reference)91.46%

Something useful beyond this project, from the first report: with the same libtorch 2.10, ROCm 7.0 and 7.1 segfault on gfx1151 and run about 5x slower than 10.2 on gfx1201. The full version matrix (7.0 to 10.2, both cards) is in the issue.

Thanks to dhekimian and DhruvDh, whose reports are in the issue. Still looking for: other cards, anyone who can reproduce the OLMo failure on gfx1100, and anyone with two AMD cards for a first multi-GPU run.


r/ROCm • • 1d ago

R9700 + 7900XTX + 7900 XT or 2 x R9700?

Thumbnail reddit.com
0 Upvotes

r/ROCm • • 2d ago

PART 2: 2x RX 7900 XTX + Qwen3.8-Flash-Next — EXL3, hipfire, Avalon, the rdna3 fork's NaN found, and every config quality-tested (spoiler: above ~3 bits it's a plateau)

28 Upvotes

A week ago I posted engine speeds for Flash-Next on my 2x 7900 XTX box. I got several interesting responses for more things to try, and software moves so fast in this space that several updates also came out, so here are the updates.

(Thanks especially to u/nasone32 (EXL3 fork), u/zeno_g2 (hipfire and others), u/sergitto11 (the #879/ROCm lead), u/xyzzing (the optimization ledger), and everyone who posted their own numbers.)

Same box: Ryzen 7 5800X, 64 GB DDR4, 2x RX 7900 XTX at PCIe 4.0 x8, now on ROCm 10.1. The speed bench is unchanged: 112 / 3.5k / 34k-token prompts, 512 out, temp 1.0 / top-p 0.95 / top-k 20, thinking on, MTP on, medians.

Critical note: quality is always checked, not just speed

Every engine and quant below also ran my private coding-agent benchmark: real bugs mined from ~10 years of my own git history, with hidden tests (results public at lab.kylebaker.io). I compare configs problem by problem on the same problems, not leaderboard averages, and use a sign test on the problems where they disagree. Since late September that's ~8,000 scored local runs and ~2,100 agent-hours. this is super slow and painful; my computer is pretty much running agentic coding benchmarks 24/7 trying to test all the models and engines and occassionally harness expierments.

Also critical: I'm not benchmaxing.

My speeds use the model's recommended sampling (temp 1.0, thinking on), not greedy, which flatters speed. The table is a fixed-prompt bench; the wall-time comparisons come from real multi-turn agent runs on hard bugs.

What that shows:

  • Above ~2.5–3 bits, Flash-Next quants are a plateau on agentic coding.
    • GSQ-RCO IQ3_XXS (76 GB), IQ3_S, UD-IQ4_XS, UD-Q4_K_XL (111 GB) and EXL3 4.05 are all within noise of each other.
    • Two runs of the same config disagree on about as many problems as two different quants do.
    • The only quant with a measurable loss is IQ1_M.
    • I thought about running q8 on the cloud to verify this, but it would cost me somewhere between $30 and $120... not worth it to me, unfortunately, as it's purely a theoretical question. (A full run on my bench suite tends to require roughly 5M generated tokens.)
  • "Does Strata trade quality for speed?": I see no sign of it. Strata's IQ3_XXS is my strongest Flash-Next config, ahead of every llama.cpp Flash-Next config I've run.
    • The two quality-trading flags u/nasone32 mentioned (STRATA_ROUTE_PRIOR, STRATA_ROUTE_TAIL_SKIP) are off on gfx1100. ROUTE_PRIOR is opt-in, and the rdna3 fork explicitly sets TAIL_SKIP=0.
    • But TAIL_SKIP is on by default on CUDA and on RDNA4 (gfx1200/1201) builds, so check yours if you compare quality on those.
    • Note that the RDNA3 fork of strata DOES seem to trade some quality for speed in my testing! but strata's iq3xxs and iq3s have both performed within noise ("slightly better", but within noise) of UD q4 k xl flash next on llama.
  • On 97 problems both ran, Strata (GSQ IQ3_XXS) solved 82 and llama.cpp (UD-Q2_K_XL) 79, with 10 vs 7 problems solved only by one side: noise.
  • On the one file I ran on both engines (UD-IQ4_XS), Strata solved 6 of 6 and llama.cpp 5 of 6. Small, but pointing the same way.

Results

engine (2x XTX unless noted)quantdecode t/s (112 / 3.5k / 34k)prompt t/s (3.5k / 34k)quality vs upstream (paired)
Strata 0.2.13.1GSQ IQ3_XXS79 / 100 / 841,440 / 2,496baseline
Strata-rdna3 fork, NaN-free variantGSQ IQ3_XXS~95–125 (119–127 in agent runs)2,418 / 3,346probably 2–5 pts weaker on the hardest problems (still testing)
EXL3 fork, 4.05 bpwEXL326 / 31 / 29686 / 1,122tie (188 vs 193 of 222; fork 183)
EXL3 fork, 3.05 bpwEXL349 / 50 / 50876 / 1,34919 vs 21 of 26 (noise)
Avalon 1200934a7, 1 cardAvalon IQ3_KS87 / 90 / 901,051 / 2,385not run at scale
Avalon 1200934a7, 2 cards, LLAMA_STAGES=0 + splitAvalon IQ3_KS64 / 75 / 641,741 / 2,434not run at scale

Notes

  • rdna3 fork NaN (#879), found. The lead that ROCm's runtime caused it didn't hold on my box: same NaN rate on ROCm 7.2.4 and 10.1. Bisecting with a replay of the requests that NaN'd pinned it to the fork's refill settings (Willing-Cover2914's reinstall fixing it is consistent with that).
    • With the 10 refill settings removed: zero NaNs across ~300 real agent runs.
    • It matches the the same refill machinery as the fork's issues #3 and #14.
    • I'm narrowing it to the exact key(s) now and will send the fork a PR soon.
    • Claude's write-up: lab.kylebaker.io/blog/hunting-a-nan-in-strata-rdna3
  • The fork vs upstream on quality.
    • Over 222 paired problems, upstream solved 193 and the fork 183.
    • I re-ran the 30 problems where they disagreed, fresh on both engines in the same week: upstream 22, fork 16.
    • Combined, about a 2% chance that's luck. So the rdna3 fork looks a few points weaker on the hardest problems.
    • While it looks dramatically faster, a weaker model writes more tokens... net: it's still ~16% less wall time per run. More paired runs are queued to pin down the size of the gap. But, again, at this point it's within 98% certainty that it's a few points weaker than upstream strata.
  • EXL3 (u/nasone32).
    • While in theory the 4.05 bpw EXL3 flash next quant should be even better than a traditional q4 quant... in my bench, it within-noise-ties IQ3_XXS on quality. IT does write ~28% fewer tokens, which suggests it may be stronger, but on 64 GB it can't keep 4.05 fully resident, so it runs at ~2.3x the wall time
  • This one was particularly interesting to me, as I was wondering if this would 'break the plateau', in theory this should be the 'strongest' flash next quant I can realistically run on my machine. Alas--given that I didn't actually see any real-world improvement, I can't say the 2.3x slower runtime (on my box) is worth it.
  • Likewise, ran the 3~bpw EXL3, still significantly slower than strata (though much faster than the 4.05bpw). Still all within noise on strength/solves on my benchmark. - Tip for agent use: set --conversation-cache-mib. It defaults to 0, and with sub-agents alternating conversations the engine re-read the whole ~150k-token prompt on every switch: ~30 s each, versus ~2 s with a 16 GB cache. That fixed a multi-hour time-out.
  • hipfire (u/zeno_g2). On the dense 27B it's shockingly good:
    • on one XTX, about 2x llama.cpp's speed: decode 94–98 vs 45–63 t/s, prompt reads ~2.1k vs ~0.9k;
    • no quality loss I can detect: 15 vs 17 of 23 paired, in about half the wall time; 60 more problems are running now.
    • Flash-Next on hipfire didn't work out on 64 GB: its 2-bit build only runs on one card, at ~15 t/s. Sad.
  • Haven't tried radiance or caminus yet.
  • Avalon.
    • Current main fixed the prompt-ring regression and the 32k out-of-memory I reported.
    • Two cards with LLAMA_STAGES=0 + manual split is the setup that works for me.
    • I'm re-checking default pipeline stages on the latest main, since issue #55 reports it working well on the same hardware.

The rdna3 fork's kernel switches, patched onto upstream Strata. The rdna3 fork's speed has two parts: tensor parallel across both cards and a table of gfx1100 kernel switches, mostly for reading prompts. I turned on 15 of those switches on upstream Strata 0.2.13.1:

  • Bench prompt reads roughly doubled: 2,549 / 4,886 t/s at 3.5k / 34k tokens, vs 1,440 / 2,499 plain. Decode unchanged.
  • Real agent runs: no wall-time gain. Prompt reading is a small share of an agent run, and cached prefixes mean only the new tokens get read.
  • Quality: 18 vs 21 solved on the same problems. Slightly worse, but within noise.
  • Strata 0.2.14+ now ships its own prompt kernels by default, so this is mostly moot; I'm testing 0.2.16 now (speed plus a quality pairing).

of note, contradicting claude's earlier assumption:

Our own first bisect round shows that with all the fork's tuning switches off and tensor parallel kept, decode dropped 23%. That's roughly back to upstream strata speed. That means most of the rdna3's fork's speed comes from its tuning switches, not from tensor parallel.

7900 XTX vs 3090?

I think dual 7900xtx is comparable to dual 3090 right now, which is kind of wild. Can anyone sanity check me on this? I bought my xtx's at about half the price of what 3090s were going for on facebook marketplace a few months ago (about $700~ vs. the about $1300~ 3090s were going for)

  • Against the best public dual-3090 Flash-Next recipe, on its own terms (greedy, thinking off): 2x XTX with the rdna3 fork matches or beats it on code and file edits (up to +31%), is 2–9% behind on prose, and reads long prompts at the same speed.
  • Watch out for greedy numbers: greedy flatters code by ~1.6x on this setup. Converted to normal sampling, I estimate the dual 3090 at ~90 t/s code and ~105–112 prose, against the fork's 104–111 and 95–109. Roughly a tie, with the XTX ahead on code.

How fast this moved

Two months ago, before strata, Flash-Next on this box meant llama.cpp on one card with experts streaming from RAM: ~15–20 t/s decode. Upstream Strata does 80–100 t/s (about 5x, no quality cost); the fork 105–125 (6–8x, a few points weaker on the hardest problems). For reference, llama.cpp on both cards does 31–36 t/s on the same bench today.

At this point RAM is my bottleneck; I'd be more likely to spend $700 floating around on more ram than another gpu.

Still testing more things, always: Strata 0.2.16 (all-own kernels, speed plus a quality pairing), the fork PR, more fork-vs-upstream paired runs, and Ternary-Bonsai-27B (5.9 GB, 1.58-bit) as a reference. If you're running radiance or caminus on RDNA3, I'd love numbers.


edit:

Strata 0.2.16 passed its speed and sanity check. Both fresh starts were clean against a same-session 0.2.13.1 baseline:

  ┌─────────────────────────┬───────────────────────┬─────────────┐
  │  2 XTX, IQ3_XXS, 262k   │      prompt read      │   decode    │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 112-token prompts       │ +14% (282 vs 248 t/s) │ +4%         │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 3.5k                    │ +20% (1,724 vs 1,432) │ +8%         │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 34k                     │ +11% (2,777 vs 2,498) │ −3% (noise) │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 256 / 512-token prompts │ +22% / +32%           │ ±, noisy    │
  └─────────────────────────┴───────────────────────┴─────────────┘
  • Correctness: both known-answer checks were right on every start, with no garbage.
  • Memory: it fits about 1,000 more experts in VRAM than 0.2.13.1 (18.8 vs 17.3 GiB of expert cache).
  • Quality pairing: its 26-problem pairing (three items) started automatically and is running now, with its 5-minute check armed.

Avalon stuff is more complicated, I still consider that project a WIP for my machine but am happily contributing my findings upstream as I find them and have high hopes for it going forward.


edit 2: Avalon, and a summary table


  ┌─────────────────────────────────┬───────────────────────────────┬─────────────────────────┬─────────────────────────────────────────────┐
  │  engine (two XTX unless noted)  │ decode t/s (112 / 3.5k / 34k) │ prompt t/s (3.5k / 34k) │                   quality                   │
  ├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
  │ upstream Strata 0.2.13.1        │ 74–79 / 75–100 / 77–84        │ ~1,440 / ~2,500         │ the reference                               │
  ├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
  │ Strata 0.2.16                   │ 77 / 81 / 75                  │ 1,724 / 2,777           │ being checked now (first item 11/12)        │
  ├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
  │ Avalon, one card                │ 86 / 92 / 91                  │ 974 / 2,768             │ untested                                    │
  ├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
  │ Avalon, two cards, #55 settings │ 87 / 103 / 98                 │ 1,718 / 2,772           │ untested                                    │
  ├─────────────────────────────────┼───────────────────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
  │ rdna3 fork, REFILL_SDMA off     │ 120–128 / 105–111 / 98–107    │ ~2,530 / 3,100–3,470    │ probably 2–5 pts weaker (workaround config) │
  └─────────────────────────────────┴───────────────────────────────┴─────────────────────────┴─────────────────────────────────────────────┘

avalon is running better, still doing tests. Hopeful that Avalon would maintain the quality of strata upstream while getting some of the speed the rdna3 fork of strata brings.

Another concern: does the newest strata, which gains some speed, do so at the expense of quality? that will take, yet again, a lot of tests.

sometimes I wish this space would calm down, I never get to the end of testing, there's always more improvements. Hard to complain about it.


edit 3:

tldr, avalon shows no quality loss like the rdna3 fork of strata, and 0.2.16 also looks like it got a speed boost with no quality loss. hipfire, further testing to validate is also still holding at showing no quality loss.

(Very promising initial results! Not statistically significant--there is likely some variation, but it may only be detectable with far more testing or on a more granular set of tests. This just means that quality may be better or worse, but it isn't a huge move in either direction--if I had any given engine trying to fix a bug, I wouldn't notice it being any worse, I'd say, at this stage.)

Results completed yesterday and today:

┌───────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────┐
  │           test            │                                          result                                          │
  ├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
  │ Avalon quality (one card, │ singles 22/24, against Strata 21 (0.2.13.1) and 22 (0.2.16). Composites: 7/12 and 8/9    │
  │  its IQ3_KS file, 26      │ bugs fixed, against Strata averages of 6.2 and 6.9. No sign of a quality cost.           │
  │ problems)                 │                                                                                          │
  ├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
  │ Strata 0.2.16 quality     │ singles 22/24, matching 0.2.13.1. Composites 5/12 and 3/9 (one run spun up 6 sub-agents  │
  │ (same 26)                 │ and hit the cap). No quality cost seen.                                                  │
  ├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
  │ hipfire vs llama.cpp,     │ L3: 17 vs 18 of 33 paired (noise) at half the wall time (7.2 vs 14.7 h). L2: only 4      │
  │ Qwen3.8-27B, round 2      │ paired so far                                                                            │
  ├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
  │ Task-driver plugin, pilot │ 12 of 27 bugs fixed, against ~15.8 in earlier runs; all 3 composites hit the 4 h cap, at │
  │  (27B Q4, laptop)         │  ~2x the tokens. The model marked about half its tasks done while their bugs still       │
  │                           │ failed. The plain arm, the real comparison, was interrupted before it finished.          │
  └───────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘

r/ROCm • • 1d ago

NEED HELP for TTS model, i have RX 6600 8gb VRAM

1 Upvotes

what should i do? as far as i have researched i can't use QWEN 3 TTS or any good one. anyone please tell me what should i do?


r/ROCm • • 2d ago

Strix Halo 2x Nightlies build

4 Upvotes

so i was benchmarking my own inference code for strix halo https://github.com/Legendofzaday/halodiffuse against comfyui and found out that rocm nightlies v2 wheels are twice as fast as the stable wheels on sd3.5 large turbo this was the case:

halodiffuse: 4.6s
comfyui nightly v2 wheels: 5.6s
comfyui stable: 11.8s

cold start times have also improved a lot:

halodiffuse: 9.1s
comfyui nightly v2 wheels: 22.6s
comfyui stable wheels: 31.2s


r/ROCm • • 2d ago

Curation vs Compaction for agentic work on budget hardware

Thumbnail
reddit.com
1 Upvotes

r/ROCm • • 3d ago

AMD ROCm 10.2 Released With Latest Optimizations, Expanded Ryzen Laptop Support

Thumbnail
phoronix.com
79 Upvotes

r/ROCm • • 2d ago

Just Purchased RX 7900 XTX 24 GB

Thumbnail reddit.com
8 Upvotes

Any suggestions related to models are welcome.


r/ROCm • • 2d ago

Four engines, seven models, one Radeon AI PRO R9700: local LLM benchmarks

Thumbnail reddit.com
2 Upvotes

r/ROCm • • 2d ago

DL research on AI 395/495 in PyTorch?

3 Upvotes

I'm not sure if this is the right place to ask this, since it seems like a lot of people are trying to run local LLMs, but can't hurt to ask.

Has anyone had any experience doing DL research using PyTorch on one of these AI 395/495 dev boxes (any brand)? AMD puts out their own version and they're are a number of vendors with their own versions. I'm wondering how stable it is. These are dramatically cheaper than the DGX Sparks, but I don't want to pull out what little hair I have left on top of my head trying to get it to run.

I would develop in Linux or WSL (don't care which). But I think I read you can't even do training under windows (not that I would do any serious research in Windows) so that suggests the drivers are still pretty immature and I should probably just pony up the $2000-3000 extra for a spark.

Thanks. If there's a better place to ask, please let me know.


r/ROCm • • 3d ago

3 x AI 9700 build - 96gb VRAM

Thumbnail gallery
9 Upvotes

r/ROCm • • 3d ago

My AMD V620 32GB is running solid after the power mod you guys recommended! Really appreciate the help! 🙌

Post image
4 Upvotes

r/ROCm • • 3d ago

How to Underclock or Undervolt AMD V620 32GB?

Post image
4 Upvotes

r/ROCm • • 3d ago

[Help Wanted] Testing GPU detection & TFLOPS benchmark paths (NVIDIA/AMD) for my from-scratch SLM trainer

0 Upvotes

Hi guys,

I’m a 15-year-old developer building SLM-CREATOR—a from-scratch, hardware-aware trainer for small, hyperspecialized text models (capped at 100M parameters) in Python and Rust. Instead of using high-level wrappers or fine-tuning existing models, I’m hand-coding the entire pipeline (detection, sizing, BPE tokenizer, and training loop) to learn how everything works under the hood.

Project Link: github.com
Tracking Issue (#1): github.com

Where I need help:
I’m developing this on a CPU-only machine right now, so all of my GPU detection and benchmarking code has only been tested against mock CLI outputs. Since r/llmdevs folks run all sorts of local setups—multi-GPU rigs, hybrid laptop GPUs, and ROCm environments—I’d hugely appreciate it if anyone with an NVIDIA or AMD GPU could run architecture.py and see if it breaks or misidentifies anything on real hardware.

What the script is doing under the hood:

  • Vendor CLI Parsing: Detects NVIDIA (nvidia-smi), modern AMD (amd-smi), or legacy AMD (rocm-smi/rocminfo) to extract architecture, total VRAM (MiB), device index, and GPU name.
  • PyTorch Index Mapping: Sets CUDA_DEVICE_ORDER=PCI_BUS_ID before importing torch and matches vendor tool names against torch.cuda.get_device_name() so multi-GPU systems don't target the wrong device.
  • Matmul TFLOPS Benchmark: Runs a 12-second thermal soak (to get past laptop boost clocks), then takes the median of 10 runs of 20 4096x4096 matmuls. It checks torch.cuda.is_bf16_supported(including_emulation=False) (or compute capability >= 8.0) so it benchmarks native bf16 and falls back to fp32 rather than getting fooled by slow software-emulated bf16.
  • Deterministic Sizing: Pipes VRAM, training hours, TFLOPS, and token counts into a Rust binary (sizer_core) to compute the max parameter count across memory, Chinchilla compute-time, and data limits.

How to test it (takes ~2 mins):

  1. git clone https://github.com
  2. Set up a venv and install requirements.txt (with your GPU's PyTorch build).
  3. Build the Rust sizer: cd sizer_core && cargo build --release
  4. Run python architecture.py from the root folder.
  5. Paste your GPU model, OS, and the terminal output (or any tracebacks) in Issue #1 or in the comments here.

I'd love any feedback on CLI parsing edge cases, ROCm quirks, or how the thermal soak behaves on your card!


r/ROCm • • 4d ago

Used RX 6800 XT vs 6900 XT (XFX Merc) for local LLMs – is the ~$63 premium worth it?

Thumbnail reddit.com
0 Upvotes

r/ROCm • • 4d ago

Calling all RX7900(gfx1100) class card owners!!!

Thumbnail
reddit.com
12 Upvotes

r/ROCm • • 4d ago

Handing two models at once in Radiance

Thumbnail
reddit.com
3 Upvotes

r/ROCm • • 5d ago

MoE Offloading update Exllamav3 on 9070XT

Post image
8 Upvotes

r/ROCm • • 5d ago

Two video models for the R9700, pick yours: LTX-2.5 or MiniMax H3

15 Upvotes

We have two new video packages out for the R9700.

Both run natively from the prompt to the mp4, and both make the sound along with the picture. They're good at different things, so here's the quick version to help you choose.

LTX-2.5MiniMax H3 v1.2
You give itA prompt, or a prompt plus a first frameA prompt plus up to 9 reference images, or a reference video with its soundtrack
Sizes1280x704, 1920x1088864x480, 1344x768
5 s clip on one R970018.1 s at 704p, 48.8 s at 1080p30.4 s with 1 image, 44.3 s with 9 (480p)
Against stock, same card7.1x and 8.6x faster2.2x to 3.8x faster
Peak VRAM27.4 GiBunder 27 GiB
First download38 GB51 GB

LTX-2.5 if you start from a prompt

Lightricks' 22B audio and video model. Type a prompt, wait 18 seconds, get 5 seconds of 704p video with sound. 1080p takes 48.8 s, and starting from your own first frame adds about 3 s.

The stock pipeline takes 128.6 s and 420.5 s for the same two clips. To be fair to it: it runs BF16, which doesn't fit in 32 GB, so it offloads to the CPU, and that's a good part of the gap.

Quality-wise we're as close to the BF16 output as Lightricks' own FP8 mode at 704p, and 1.1 dB PSNR further away at 1080p.

What this first build can't do yet: portrait sizes, sizes other than the two above, and it needs the clip length up front.

MiniMax H3 if you want your own subjects in it

Give it a photo of your dog, your product or yourself, write "the dog from <Picture 1> runs through a sunlit forest", and that's the dog you get. Up to nine images, or a video whose soundtrack it picks up. Audio is 32 kHz stereo.

v1.2 replaces the ComfyUI package we shipped in October. Everything runs on our HIP kernels now, with no PyTorch, Triton or ComfyUI in the container. The more references you give it, the bigger the win over the stock engine:

5 s clip, Turbo4Stockv1.2
864x480, 1 image67.4 s30.4 s
864x480, 3 images91.2 s33.4 s
864x480, 9 images167.5 s44.3 s
1344x768, 9 images278.7 s114.1 s

Stock here is MiniMax's ComfyUI workflow with its own engine, same prompts, images and seed, running the W4A8 checkpoint from our old package. With nine images it spends over half its time just encoding them, and that's where v1.2 gains the most.

Limits: up to about 9.4 s of video at 768p with nine images, no audio-only references, and the 50-step Base preset is still on v1.1.0.

Try one (or both, one at a time)

They don't fit on the card together, so stop one before you start the other. One R9700, Docker and a ROCm driver:

git clone --depth 1 https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin


# LTX-2.5 (accept the license on the HF page first, then use a token from that account)
export HF_TOKEN=<your token>
./models/LTX-2.5/launch.sh
# second terminal, after READY:
./models/LTX-2.5/run.sh generate "A street drummer plays a fast rhythm on a city sidewalk at dusk while the crowd claps and cheers" --out drummer.mp4


# MiniMax H3 v1.2
PAITON_DETACH=1 ./models/MiniMax-H3/native/launch.sh http
docker logs -f paiton-minimax-h3-v12   # wait for READY
curl -o panda.mp4 -F image=@panda.png \
  -F prompt='The red panda from <Picture 1> walks through a sunlit forest. Audio: birds and a stream.' \
  http://127.0.0.1:30000/v1/videos/generations

Guides, with all the numbers and options:

Licenses: LTX-2.5 is under the LTX-2.x Community License (companies with US$10M+ yearly revenue need a paid license for anything past non-commercial testing and research), MiniMax H3 under the MiniMax H3 Community License.


r/ROCm • • 5d ago

Hierarchos-Native v0.2.0: RDNA3 WMMA, native multimodal training, and a lot of FP32 Vulkan optimization!

4 Upvotes

Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.

I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.

The new v0.2.0 release is now available!

GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

1. Native RDNA3 WMMA / cooperative matrix acceleration

One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.

The new Mixed WMMA execution arm uses:

  • FP16 matrix operands.
  • FP32 matrix accumulation.
  • FP32 master weights and gradients.
  • FP32 AdamW optimizer moments and updates.
  • FP32 normalization, softmax, and loss calculations.

The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.

I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.

Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:

WorkloadMeasured kernel speedup
QKV projection2.17×
FFN up projection2.12×
FFN down projection2.31×
Vocabulary projection1.83×

These are individual GEMM measurements, not whole-model speedups.

There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.

In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.

WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.

Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.

Documentation: RDNA3 WMMA guide

2. Native multimodal image and text training

Another major development area has been the image-text pipeline.

The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.

Some of the implemented work includes:

  • Native RGB image input and preprocessing.
  • CLIP image encoders and LLaVA projector integration.
  • SigLIP / Gemma 3 vision training.
  • Gemma 4 vision integration.
  • Joint image/text training and encoder-decoder integration.
  • A dedicated DeepSeek V4.1 image-training path.
  • Backpropagation through the image tower and projector.
  • Export, reload, and checkpoint-related validation.

This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.

For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.

The qualification work has also expanded considerably:

  • 44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
  • The Gen9 qualification retains 95 image cases and 166 text regression cases.
  • A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
  • Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.

The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.

These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.

Documentation:

3. A lot of additional FP32 optimization

I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.

This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.

Some examples:

RDNA3 FP32 work

  • Specialized attention forward/backward paths.
  • Image-text attention and projector optimizations.
  • RMSNorm, RoPE, GELU, and other arithmetic specializations.
  • Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
  • More careful matching of CPU reference arithmetic and GPU execution order.

Intel Gen9 work

  • Additional native linear and gradient kernels.
  • Attention and activation specializations.
  • More efficient parameter upload and staging behavior.
  • Continued optimization of DeepSeek V4.1 training and execution.

One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.

On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.

On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.

These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.

4. Preserving the original FP32 path

An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.

I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:

  • 12 alternating ABBA rounds per workload.
  • Five warm-up runs and 30 timed forwards per arm.
  • Separate, hashed baseline and candidate binaries.
  • Exact FP32 logit digest comparisons.

All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.

The worst paired performance ratio was 1.0007.

This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.

5. Getting started

The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.

You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.

Model weights are downloaded separately or supplied through a local model directory.

After extracting the archive, you can run:

.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-info

The final command is especially useful for checking which precision capabilities your GPU actually exposes.

Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

What's next?

There is still a lot I'd like to improve.

In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.

I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.

If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.

Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.

Repository: https://github.com/necat101/Hierarchos-Native

Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.