r/nim • u/leejangjong_bobby_de • 8h ago
Nimformers: Been learning Nim and tried writing a Transformer inference backend in it — Metal works, Vulkan still fighting me
Hey everyone!
I've been tinkering with Nim for a while, and I finally decided to try something I'd been putting off for ages: writing my own Transformer inference backend in Nim. I'm still very much learning as I go, so this is a personal project rather than anything production-ready.
The idea was to see how far I could get with a native Nim implementation that talks directly to Metal on Mac and Vulkan elsewhere, without dragging in PyTorch or any big framework. I got int8/int4/fp8/fp4 quantization working, plus NIMQ state dicts, and I've been running some benchmarks on two devices.
Being honest about where it stands:
- Metal works reasonably well on Mac, but the Vulkan path is still shaky — on Iris Plus it's noticeably slower than Metal and I haven't figured out why yet
- The APIs are still moving around, so some interfaces aren't stable
- The ecosystem is tiny, and the docs can't compete with PyTorch's
Repo is here if anyone wants to poke at it: https://github.com/bobbyshop-vui/nimformers-framework
I'm mostly curious: for those of you doing low-level / systems work in Nim, how do you usually structure the backend abstraction when you're targeting multiple GPU APIs? And is there anything obvious I'm missing on the Vulkan side that would explain the perf gap?
Would love to hear your thoughts!