A Machine Learning Approach to Estimating Energy Use in Language Model Inference

Abstract

The rapid expansion of Large Language Model (LLM) serving in cloud data centers has created a critical need for energy-aware scheduling. However, estimating inference energy typically requires hardware-level power telemetry, which is rarely accessible to cloud tenants. This paper proposes a lightweight, machine-learning-based framework for estimating the energy consumption and CO2 emissions of LLM inference using only software-observable runtime features. We collected 4,950 inference records across five model architectures in a Vertex AI environment, evaluating varying batch sizes and simulated demand levels. We demonstrate that a Random Forest regressor can accurately predict inference energy (R2 = 0.994) without requiring privileged hardware instrumentation. Feature-importance analysis reveals that inference latency accounts for 76.7% of the model’s predictive variance, far outweighing token count as the primary driver of energy cost. To demonstrate practical utility, we implement an interactive prototype for real-time energy estimation. This work establishes latency as a highly reliable proxy for energy, providing system designers with an actionable metric for sustainable LLM deployment.