r/LLMDevs • u/Maui-The-Magificent • 11m ago
Help Wanted What is the normal expected RSS on say Qwen3.5:0.8b BF16 by doing disk reads?
Hi!
I am working on a model re-representation compiler called Cuddler. I have recently finished a mid-way goal of mine, which was to run inference through a different decoder based on a geometry i built for my graphics work + a model viewer i built to read and create an atlas of the tokens associated with the frozen weights.
Today i succeeded, finally, in getting it to work. It runs on one cpu core which means its not that fast. But because the decoder is iterative, it can get away with what I hope is a low amount of data read into memory.
Anyway, for context. I am a systems engineer, and I mostly focus on fundamental systems, which translate to me both having very limited depth in abstracted work/environments/problems, I also try to stay as naive as possible about whatever domain I work in until i actually know i need to learn something about it.
In short, this has led to me have much more mechanical understanding of AI models and their architecture rather than their runtime.
So, I was hoping if you guys could share some numbers and stats on how your disk inference techniques run, especially what your rough RSS is? This way I can compare if my numbers are good or not, and in what way it would need more work. The next step is re-representing the original weights themselves.
Anyway, my numbers are a static 256mb arena in memory. I have only tested a limited amount of prompts so far, and I have measured 109mb peak RSS.
this is an image of the readout.