Phase-aware Peak Power Reduction for Minimizing the Capital Expense of LLM Inference
TL;DR
PPPR is proposed, a Phase-aware Peak Power Reduction framework for minimizing the CapEx of LLM inference that features two novel LLM-specific designs and phase-aware timing that shaves more peak power and leads to lower CapEx.
Abstract
The rapid growth of large language model (LLM) inference services has been driving data centers to quickly increase their hosted GPUs and servers, causing data centers to approach their peak power capacities. This is a serious challenge because 1) expensive upgrades of power facilities must be done to existing data centers or 2) new data centers must be constructed, both of which require a significant increase of capital expense (CapEx). Existing solutions for peak power reduction include frequency throttling and power capping, which both rely on GPU frequency throttling. Hence, when applied directly to LLM inference, they can degrade performance and increase latency. While energy storage can be exploited to reduce peak power draws without hurting performance, prior solutions are not designed for LLM inference, which has a unique and recurring two-phase power profile with short-duration, high-power prefill phases followed by longer, low-power decode phases. In this paper, we propose PPPR, a Phase-aware Peak Power Reduction framework for minimizing the CapEx of LLM inference. PPPR features two novel LLM-specific designs. First, in contrast to prior work that usually relies on preset thresholds to decide when storage should discharge or recharge, PPPR leverages LLM phase information to make better decisions. This phase-aware timing shaves more peak power and leads to lower CapEx. Second, based on real-world LLM traces, a hybrid storage architecture is designed with server-level supercapacitors to handle short prefill spikes and rack-level batteries to smooth longer, aggregated power changes. This novel placement and sizing strategy can lead to more CapEx savings for LLM inference. Our hardware evaluation with representative LLM workloads shows that PPPR achieves 1.29 × more peak reduction, on average, than three state-of-the-art baselines. Extensive trace-driven simulations also show that PPPR achieves up to 3.53 × higher CapEx savings and up to 1.52 × longer UPS lifetime than the baselines.