r/CUDA • • 3h ago

Built a ternary (BitNet-style) inference engine in Rust/CUDA: 8x faster than dense f32 on one CPU core. Looking for GPU testers

Thumbnail reddit.com
1 Upvotes

r/CUDA • • 6h ago

Sm_120 RTX 5070 causal conv1d and Ssm wheels

2 Upvotes

RTX 5070/5080/5090 (sm_120, compute 12.0) currently fails with "no kernel image is available" because setup.py only builds for 70;80;90.

I built working wheels for Blackwell:

\- Patch: TORCH_CUDA_ARCH_LIST=12.0 and replace sm_70/sm_80/sm_90 with sm_120 in setup.py

\- Tested on 5070 WSL2 + torch 2.11.0+cu128 + py3.10

\- Wheels here for anyone stuck: https://github.com/Rapa2026/mamba-ssm-5070-blackwell-wheels

Happy to PR the sm_120 fix if you want it.

Test: Mamba2(d_model=64).cuda() -> OK


r/CUDA • • 10h ago

Replaced most inline PTX with MLIR ops, cutting asm regions from 499 to 81 across 20 kernels.

8 Upvotes

While building a high-level language with a rich type system, a core goal is reducing the need for low-level escape hatches like inline asm.

In glm-vx(GLM-5.3 serving engine with kernels in Vx-lang, checkpoint loading, and streaming HTTP), we replaced most inline PTX with MLIR ops, cutting asm regions from 499 to 81 across 20 kernels.

When LLVM sees the surrounding code, it can simplify logic and cut spills thus generating better code.

e.g., A bf16 store. The epilogue of the GEMM converts each f32 result to bf16 and stores 16 bits. The asm converted into a 16-bit register and then widened it to 32 bits with cvt.u32.u16, only for the store to use the low 16 bits again. Now the store takes the 16-bit register, and fp8_gemm_m64 has 64 fewer instructions:

// before
{ .reg .b16 h; cvt.rn.bf16.f32 h, %r393; cvt.u32.u16 %r199, h; }
st.global.b16  [%rd65], %r199;

// after
cvt.rn.bf16.f32  %rs1, %r327;
st.global.b16    [%rd74], %rs1;

But when hoisting/reordering hurts (like %laneid or mma scheduling on H200), we have no option to keep the asm. Link to the full writeup in comments.


r/CUDA • • 23h ago

why it's so hard to make SASS assembler?

Thumbnail redplait.blogspot.com
14 Upvotes

summary:

  • only ~80% of ELF sections currently known
  • ~50% of EIATTRs
  • instruction scheduler is highly gpu specific and can't be implemented based on llvm machine scheduler. And np-hard

r/CUDA • • 1d ago

Tencent gave Google's Guetzli a GPU. I volunteered to keep it alive. v3.0.0 is out.

Thumbnail reddit.com
3 Upvotes

r/CUDA • • 2d ago

exact-gpu-jpeg: CUDA JPEG decode/encode that’s byte-identical to libjpeg-turbo and Pillow

8 Upvotes

GPU JPEG decoders are fast, but their pixels differ from the CPU libraries'. On 102 public test images, nvJPEG matched libjpeg-turbo on none of them, by up to 110 levels at 4:2:0. That quietly breaks reproducible ML pipelines, test baselines and caches whenever decoding moves between CPU and GPU.

exact-gpu-jpeg decodes on the GPU and returns libjpeg-turbo's pixels byte for byte (accurate IDCT, fancy upsampling), and its encoder writes the same bytes as Pillow. Anything it can't reproduce exactly, progressive files for example, goes to libjpeg-turbo itself, so the output is always the reference.

On a 2016 GTX 1070: a 30 MP camera photo decodes in 11.3 ms, against 185 ms for libjpeg-turbo on one core and 164 ms for nvJPEG on the same card. Batches of small images: 7,310 images/s. Encoding: 5.5x faster than Pillow with identical files.

Where it loses is listed near the top of the README: one small image at a time, small encodes against 8 CPU threads, progressive files. Everything so far is from one machine, so results from other GPUs are very welcome: `python bench/reproduce.py --quick`, and there's a pinned issue for them.

The parallel Huffman stage (Weißenberger & Schmidt's self-synchronising method) is drawn in docs/huffman-sync.md, and there's a white paper in docs/ covering the method, the evidence and the limits.

https://github.com/mrluker/exact-gpu-jpeg


r/CUDA • • 2d ago

My CPU won all 38 final-state tests. The RTX 5060 pulled ahead when I returned every step.

0 Upvotes

I’m working on Ohio at Primitive. I wanted to see where an RTX 5060 actually beats native C for a simple workload: update a lot of integer states and get the results back to the host.

When I only returned the final state, the CPU had the lower median in all 38 test conditions.

When I returned every step for 100,000 entities over 120 updates, the GPU pulled ahead on both seeds. On seed 97, it was 10.50 ms on the RTX 5060 with PyTorch CUDA vs 14.43 ms in native C. The outputs matched exactly. Those are warm timings with transfers included.

One thing I’d change before running it again: I returned 12 million states as int64. Every value fits in a byte, so the output could go from 96 MB to 12 MB.

If you were running that comparison, what CPU and CUDA implementations would you put side by side for the uint8 run?

[Primitive’s public GitHub](https://github.com/Primitivelabai/primitive-open). The code and timing logs for this run are still private.


r/CUDA • • 3d ago

I made an iPhone app for practicing CUDA with short puzzles, and you can run kernels on a real GPU from it

29 Upvotes

Hi,

I wanted a way to practice CUDA in short breaks, like on the bus or while an AI agent is working, so I made an iPhone app for it.

It teaches CUDA as short puzzles. You start with thread and block indexing then go on to 2D grids, warps and occupancy. Later skills are about reading PTX.

There are also labs. You fill in the missing lines of a kernel, tap Run, and it runs on a real GPU. The labs come from my main project, GPUFlight Performance Lab, a web tool for running and profiling GPU kernels. The app uses the same backend, so the report numbers come from a real profiled run.

It's free to try, iOS only for now. I'd like to know what people here think, especially whether the puzzles are too easy and which topics are missing.

https://apps.apple.com/us/app/gpu-performance-lab/id6829988955


r/CUDA • • 3d ago

Senior AI-Native Systems SWE, TensorRT

Thumbnail reddit.com
2 Upvotes

r/CUDA • • 3d ago

Anyone think it would be cool to create VUDA(CUDA) using vulkan? Its a big task I know but probably will be a proof of concept thing to begin with.

Thumbnail reddit.com
0 Upvotes

r/CUDA • • 4d ago

Update: Fixed warp32 emulation and unbounded while loops. Looking for AMD hardware to test.

1 Upvotes

A few days ago I posted about UniversalGPU, my open-source CUDA-to-x86/PTX/GCN compiler. The top comment asked: "Does --warp-size=64 preserve two separate 32-lane groups for shfl.sync, or change their semantics?"

I fixed it. The --warp-size=64 path now wraps every shfl.sync in an scf.if that only executes if the source lane is in the same 32-lane half. Here is the generated MLIR:

%lane_2 = arith.remsi %tx, %c32 : index

%half_3 = arith.divsi %tx, %c32 : index

%src_half_4 = arith.divsi %iv, %c32 : index

%same_half_5 = arith.cmpi eq, %half_3, %src_half_4 : index

%1 = scf.if %same_half_5 -> (f32) {

%shfl_6 = nvvm.shfl.sync.down %c4294967295, %0, %iv : f32

scf.yield %shfl_6 : f32

} else {

scf.yield %0 : f32

}

I also fixed 5 other compiler bugs from a stress test: modulo operator, compound assignment to memory, unique loop variables, unbounded while loops (now correctly lowered to scf.while with iter_args), and switch case stores.

What I need now: someone with an AMD GPU (MI200, MI300, or RX 7000) to test the warp32 emulation on real hardware. I don't have AMD hardware, and I can't validate the emulation without it. If you have a GPU and 10 minutes, I'd love your help.

Repo: https://github.com/7636-lochan/UniversalGPU


r/CUDA • • 5d ago

Initial VuPy release: Numpy/CuPy-like GPU array library backed by vulkan

Thumbnail github.com
3 Upvotes

r/CUDA • • 5d ago

Got placed, have 6 months before joining. Should I explore Linux kernel development, GPU programming or stick with AI/ML?

19 Upvotes

I'm a final-year engineering student in India from a non-CS background. I've recently secured an offer at an AI/analytics services company, and I have around 6 months before joining.

I've been exploring different areas of computer science, and I've realized that I'm increasingly interested in low-level programming and understanding how systems work internally.

MY BACKGROUND:

I've independently learned programming, ML/DL and worked on projects involving: computer vision , RAG , agentic AI

However, I dont want to spend my career primarily working with high-level AI frameworks and application development. because when i developed these projects , i didnt feel anything developing it , no satisaction at all

AREAS I'M CONSIDERING:

  1. Linux and Red Hat ecosystem:
  2. 2. GPU Programming:
  3. 3. Systems Programming
  4. 4. AI Infrastructure:

I find the idea of understanding what happens beneath the abstractions much more interesting than simply using existing libraries.

WHAT I'M CONFUSED ABOUT:

I have six months before joining my company, and I don't want to waste that time jumping between technologies.

I'd appreciate advice from people working in systems engineering, Linux, kernel development, GPU programming or related fields.

  1. Is low-level systems programming a good career direction in 2026?
  2. Between Linux kernel development, GPU programming and general systems programming, which would be a better starting point?
  3. Is CUDA/GPU programming a niche with good long-term opportunities, particularly with the growth of AI?
  4. What fundamentals should I master before attempting serious projects? C, C++, OS, computer architecture, networking, compilers?
  5. Would contributing to the Linux kernel or other open-source systems projects actually help me get shortlisted?

r/CUDA • • 6d ago

Bringing Data-Center-Class AI Inference to Consumer NVIDIA RTX GPUs: Porting Atlas to Ada Lovelace

Thumbnail github.com
6 Upvotes

I've spent the last several days contributing to Atlas, an open-source Rust + CUDA inference engine originally targeting NVIDIA's GB10 with 120GB of unified memory.

I wanted to see how far I could push it on my RTX 4060 Ti with just 16GB of VRAM. It turned into quite a bit more than getting it to compile.

Running Qwen3.5-4B, here's where I ended up:

  1. 381 tok/s aggregate across 8 concurrent requests (roughly 48 per user) using experimental int4 + speculative decoding.
  2. 98 tok/s for a single user with the same configuration.
  3. 4.6x faster BF16 prefill, from 794 to 3,684 tok/s. This preserves BF16 precision and is enabled by default on Ada.
  4. 2.1GB less VRAM usage with the optional FP8 feed-forward path.
  5. Successfully served a 159,807-token prompt with FP8 KV cache and retrieved a needle from it.
  6. All 223 Ada kernel compilations passing after fixing architecture incompatibilities.

One interesting finding: Nsight showed the GPU was already busy 95.6% of the time at eight streams. Several CUDA Graph and fusion optimizations I expected to help ended up making practically no difference.

For comparison, BF16 reached 195 tok/s across eight users, and FP8 FFN reached 222 tok/s.

Important caveat: the 381 tok/s configuration is lossy, and I haven't established task-level quality equivalence with BF16. These are screening measurements, not fully repeated confirmation runs.

Atlas is developed by Thomas Braun and the Avarok-Cybersecurity contributors. My work focuses on the Ada port, memory management, kernel optimization, and consumer GPU benchmarking.

The goal is to get more useful local inference out of hardware people already own.

GitHub: https://github.com/Avarok-Cybersecurity/atlas

Happy to discuss the kernel work or benchmark methodology with anyone working on Rust, CUDA, or local inference.


r/CUDA • • 6d ago

day 4 of building a GPU from scratch (32 bit)

Thumbnail reddit.com
1 Upvotes

r/CUDA • • 6d ago

vulkan-py easymode. all the boilerplate is in easy.py. vulkan-py fork https://github.com/billyblackburn/vulkan-py-easy/tree/main

Thumbnail reddit.com
2 Upvotes

r/CUDA • • 7d ago

Day 6 of Hopper kernels: DeepSeek style block scaled fp8

Thumbnail x.com
4 Upvotes

r/CUDA • • 7d ago

We replaced NVCC with our own PTX compiler. Results after a 57-entry hackathon.

29 Upvotes

Emmy is the open-source compiler for LLM inference. Originally, we emitted CUDA and fed it to NVCC, but it was sluggish; some kernels took minutes to compile with -O3, so we replaced the CUDA emitter with PTX.

TL;DR: Our own compiler + PTXAS is on par with NVCC.

The hackathon

We ran a contest: compile our internal Kernel IR to PTX, judged on hidden tests and speed against our NVCC build. 57 teams entered. The top two:

Most of their speedup came from changing the schedule: sharing work across lanes, re-tiling attention, fusing sibling loops. With those rewrites turned off, the winner’s submission matches NVCC.

Almost all teams wrote PTX directly. The one LLVM-based entry scored well (1.17×), but LLVM has no form for many of the instructions we need, so tensor-core and async-copy instructions became opaque inline asm, and compilation is slower.

Performance against nvcc

  • Timing lane on an RTX 5090 (211 kernels): median 1.00.
  • We re-timed stored kernel configurations on four GPUs (5090, H100, A100, V100): medians between 0.98 and 1.01 at first. Then we fixed the slow tail.

Fixes that mattered, in µs (nvcc / ours before / ours after):

FixCasenvccBeforeAfter
Unsigned grid-decode division with cicc's 32-bit fast path5090 attention97.647899.4
Two fp8 bytes as one 16-bit load, then a packed convert5090 NVFP4 GEMM126.6237.5126.6
Volta fragments as one 128-bit shared load, not eight 16-bit onesV100 Llama-3.1 piece14650176
cicc's unroll rule for strided lane loops, written in Kernel IRA100 softmax18.629.817.9
.pragma "nounroll" on loops we already unrolled5090 SDPA (ptxas time 0.53 → 0.06 s)8819489
.minnctapersm 1 on single-wave launches5090 matmul–18.310.9
A predicate kept for a bool used as a number5090 MoE router43.964.244.1

What nvcc does that we had to copy

  • It's unroll rules. cicc unrolls fully up to a size budget and makes two copies of some loops, and ptxas then unrolls those further. We write all of that in Kernel IR and mark every unrolled loop nounroll. If you leave a compact loop unmarked, ptxas unrolls it fully, with spills and seconds of assembly time.
  • It's pointer stepping. In C, signed overflow is undefined, so cicc turns base + i*4 with an int i into a 64-bit pointer stepped each trip. PTX arithmetic wraps by definition, so ptxas can’t assume that. We keep a 64-bit copy of the loop counter so ptxas can fold each unrolled copy into [base + constant]
  • ld.global.nc for inputs; the kernel never writes.
  • Its expf sequence.

Where we're ahead or behind

  • Ahead: V100 Llama-3.1-8B pieces up to 1.9× (76 vs 146 µs). H100 512 matmul 10.3 vs 11.5 µs. Six 5090 lane cases are more than 5% faster.
  • Behind: one V100 GEMM 1.11×. H100 FP32 matmul with a dynamic size 1.08×. A handful of kernels 1.1 to 1.2× slower where PTXAS picks a different load schedule from nearly the same PTX. Every respelling we tried traded one case against another.

Compile time (RTX 5090, CUDA 13.0, 269 kernels)

Median per kernelp90Whole set
Our compiler (Kernel IR to PTX)0.9 ms4.9 ms
ptxas12 ms48 ms
Ours total~13 ms
nvcc --cubin173 ms334 ms

About 10× faster.

WIP: https://github.com/cloudrift-ai/emmy/pull/1399


r/CUDA • • 8d ago

[Help Wanted] Testing GPU detection & TFLOPS benchmark paths (NVIDIA/AMD) for my from-scratch SLM trainer

Thumbnail reddit.com
0 Upvotes

r/CUDA • • 8d ago

CUDA vs. Everything Else

Thumbnail youtube.com
1 Upvotes

Right or Wrong? GPU better than CPU?


r/CUDA • • 8d ago

Day 5 of writing Hopper kernels everyday - Clusters and TMA multicast

Thumbnail x.com
3 Upvotes

r/CUDA • • 9d ago

should i make a dataframe and numpy package for wgpu-py. i have it wotking on my amd gpu?

Thumbnail reddit.com
0 Upvotes

r/CUDA • • 9d ago

TrainStream: LoRA on Qwen2.5-14B in 0.53 s/step on a 24 GB laptop GPU, faster than bitsandbytes and Unsloth in my tests. One line of code, no DeepSpeed or bitsandbytes needed.

0 Upvotes

I've been building TrainStream, a C++/CUDA extension for PyTorch for training and fine-tuning models that don't fit in your GPU's memory. It moves optimizer state, gradients, activations and, if needed, the weights themselves into system RAM, then streams each one back over PCIe just before the GPU uses it.

import trainstream
model, optimizer = trainstream.wrap(model, optimizer)

It has no third-party dependencies, only the C++ standard library, the CUDA runtime and PyTorch's ATen. It doesn't use DeepSpeed, bitsandbytes or CUTLASS.

What it does

  • Offloading. Optimizer state goes to a CPU AdamW whose updates match torch.optim.AdamW (max difference ≈3e-7 after 20 steps). Activation and weight paging are bit-exact.
  • Quantized frozen bases. A LoRA base can be stored as int8, NF4, FP8 or NVFP4. FP8 and NVFP4 run directly on Blackwell tensor cores (RTX 50-series, B200).
  • Fused kernels. RMSNorm, RoPE, SwiGLU/GeGLU and LoRA (2 matmuls forward, 3 backward per adapted Linear).
  • Automatic tuning. It times recompute against streaming for activations and picks the fastest attention backend for your shapes. It also logs every decision it makes, so you can see why.
  • Extras: padding-free sequence packing, DPO without a second model in memory, a paged KV cache for generation during RL, 8-bit optimizer moments, a disk tier for weights that don't fit in RAM, LoftQ init and a Hugging Face Trainer integration.

Numbers (RTX 5090 Laptop GPU with 24 GB, WSL2, Qwen2.5-14B, LoRA r=16, median of 10 steps after warm-up)

1×5121×2048
TrainStream, NVFP4 base0.530 s1.57 s
TrainStream, NF4 base0.609 s2.17 s
bitsandbytes QLoRA + PEFT0.687 sspills past VRAM
Unsloth (4-bit)0.778 s2.73 s
bnb QLoRA + Liger0.909 s2.88 s

Full fine-tune of Qwen2.5-1.5B (batch 4×512): 1.23 s/step at 18.8 GB, against 1.85 s for stock PyTorch (which peaks at 32 GB and spills) and 2.12 s for DeepSpeed ZeRO-2 with offload.

Caveats

  • Every number comes from one laptop (WSL2 and Windows 11). Native Linux, desktop cards and multi-GPU machines are untested.
  • Held to 12 GB of VRAM, it ties Unsloth (0.834 s vs 0.819 s, within run-to-run spread). Most of the speed lead comes from using the VRAM it has.
  • NVFP4 is fast but costs quality. It adds about +5 % perplexity on 14B (+11 % on 1.5B), against +3 % / +6 % for NF4, because it quantizes activations as well as weights. It's opt-in, and the README has the full perplexity table.
  • The benchmark harness runs close comparisons 3 times and counts a win only when the gap is wider than the spread. It's included, so you can rerun everything.

In progress, not released or benchmarked yet: a NaN/Inf guard that skips bad steps cleanly, PiSSA init, unbuffered async disk reads (io_uring / IOCP), background checkpoints, MoE expert paging, FP8 gradient transport and a TCP ring all-reduce.

Install: pip install trainstream (or uv). You need CUDA PyTorch, the matching CUDA toolkit and a C++ compiler, because the extension compiles on first import.

Apache 2.0: https://github.com/johnanishg/TrainStream

I'd really like to hear from anyone who runs it on other GPUs (especially Ampere/Ada desktop cards or native Linux), and from anyone who can poke holes in the benchmark method.