r/EnergyAndPower • u/oguzhanssan • 4h ago
Why vLLM burns peak 700W during autoregressive decode (and our open-source C++ NVML pacer to fix it)
Hey everyone,
If you run vLLM or Triton on modern accelerators (H100, A100, L40S, etc.), you probably noticed that GPUs stay locked at maximum core clock frequencies (e.g., 1980 MHz on H100) through the entire generation step.
### The Problem: Silicon Asymmetry in Serving
- **Prompt Prefill:** Compute-bound (GEMM). Arithmetic intensity is > 100 FLOPs/Byte. You *need* peak SM clocks to keep TTFT low.
- **Autoregressive Decode:** Memory-bound (GEMV). Arithmetic intensity drops to ≤ 4 FLOPs/Byte. Over 85% of step time is spent stalling on High-Bandwidth Memory (HBM) reads.
Because dynamic CMOS power dissipation scales quadratically with voltage ($P_{dyn} \propto V^2 f$), sustaining peak core clocks while memory stalls burns massive amounts of electricity without improving token latency.
### What we built: TensorPackAI (Community Edition)
We just open-sourced **TensorPackAI**: a zero-overhead C++17 NVML runtime layer with Python bindings:
- **Fast NVML Direct Loader:** Direct `dlopen` of `libnvidia-ml.so` (eliminates standard python ctypes latency).
- **Prometheus Telemetry:** Exposes windowed `tensorpack_joules_per_token` and `tensorpack_tokens_per_watt` on port `:9100/metrics`.
- **CLI Tuner (`tensorpack-tune`):** Inspect real-time per-GPU power and recommend memory-paced decode clocks.
### Quick Test:
```bash
pip install tensorpackai
tensorpack-tune status
```
In our Chatbot Arena benchmark runs, pacing core clocks to memory speeds during decode drops dynamic power by up to **58.1%** while keeping P90 Inter-Token Latency (ITL) strictly under human-perceptible thresholds (<20ms).
- **GitHub (Apache 2.0):** https://github.com/tensorpackai/tensorpackai-community
- **Docs:** https://tensorpackai.com/docs
Would love to hear your thoughts, feedback, and edge cases on different architectures (especially A100/L40S)!