Fine-Grained Energy Accounting in Production LLM Serving
TL;DR
KV (Key-Value) volume is introduced, a physically grounded metric that captures the spatiotemporal footprint of a request’s KV cache occupancy, and it is shown that energy per KV volume (EPV) provides a stable and reproducible signature for modeling serving energy.
Abstract
Production large language model (LLM) serving uses continuous batching to maximize GPU utilization. While shared compute and memory resources improve throughput, they entangle the energy footprints of concurrent requests and make per-request energy physically unobservable. It is thus critical to develop a fair method for disaggregating device-level energy to individual requests. We tackle this challenge by developing a new approach to energy accounting in LLM serving. We introduce KV (Key-Value) volume, a physically grounded metric that captures the spatiotemporal footprint of a request’s KV cache occupancy, and show that energy per KV volume (EPV) provides a stable and reproducible signature for modeling serving energy. Building on EPV, we develop a state-aware energy model and a game-theoretic attribution method, and further build WattsOnLLM, a lightweight system for practical online energy accounting in existing LLM serving platforms with request-level granularity. We demonstrate that conventional heuristics deviate from the Shapley fairness target by 51.6% to 95%. In contrast, WattsOnLLM achieves a macroscopic energy estimation error as low as 1.9% and approximates the Shapley fairness target within 5.25% with very low overhead.