r/EnergyAndPower • • 23h ago

Electricity Bill

Post image
2 Upvotes

r/EnergyAndPower • • 15m ago

Why vLLM burns peak 700W during autoregressive decode (and our open-source C++ NVML pacer to fix it)

2 Upvotes

Hey everyone,

If you run vLLM or Triton on modern accelerators (H100, A100, L40S, etc.), you probably noticed that GPUs stay locked at maximum core clock frequencies (e.g., 1980 MHz on H100) through the entire generation step.

### The Problem: Silicon Asymmetry in Serving

- **Prompt Prefill:** Compute-bound (GEMM). Arithmetic intensity is > 100 FLOPs/Byte. You *need* peak SM clocks to keep TTFT low.

- **Autoregressive Decode:** Memory-bound (GEMV). Arithmetic intensity drops to ≤ 4 FLOPs/Byte. Over 85% of step time is spent stalling on High-Bandwidth Memory (HBM) reads.

Because dynamic CMOS power dissipation scales quadratically with voltage ($P_{dyn} \propto V^2 f$), sustaining peak core clocks while memory stalls burns massive amounts of electricity without improving token latency.

### What we built: TensorPackAI (Community Edition)

We just open-sourced **TensorPackAI**: a zero-overhead C++17 NVML runtime layer with Python bindings:

  1. **Fast NVML Direct Loader:** Direct `dlopen` of `libnvidia-ml.so` (eliminates standard python ctypes latency).
  2. **Prometheus Telemetry:** Exposes windowed `tensorpack_joules_per_token` and `tensorpack_tokens_per_watt` on port `:9100/metrics`.
  3. **CLI Tuner (`tensorpack-tune`):** Inspect real-time per-GPU power and recommend memory-paced decode clocks.

### Quick Test:

```bash

pip install tensorpackai

tensorpack-tune status

```

In our Chatbot Arena benchmark runs, pacing core clocks to memory speeds during decode drops dynamic power by up to **58.1%** while keeping P90 Inter-Token Latency (ITL) strictly under human-perceptible thresholds (<20ms).

- **GitHub (Apache 2.0):** https://github.com/tensorpackai/tensorpackai-community

- **Docs:** https://tensorpackai.com/docs

Would love to hear your thoughts, feedback, and edge cases on different architectures (especially A100/L40S)!


r/EnergyAndPower • • 3h ago

Looking for an article

Thumbnail sciencedirect.com
0 Upvotes

[Article] Artificial intelligence based forecasting and optimization model for concentrated solar power system with thermal energy storage