Uncore Frequency De-Tuning for Power Efficient LLMs at Telco Edge

Abstract

A key pillar of the evolving telco cloud landscape is the potential hosting of AI workloads such as Large Language Model (LLM) inferencing instances. While Graphics Processing Units (GPUs) are often favoured for best possible performance, it is also feasible to run certain LLM models on more standard Central Processing Unit (CPU) based compute infrastructure. Indeed, multiple LLMs can run in a “multi-tenant” format, as long as suitable resource isolation mechanisms (particularly CPU core pinning/affinity) is enforced. Within an edge compute environment where resources are generally more constrained, including the ready availability of power, it is crucial to achieve a balance between attainable “good enough” performance of LLMs, while also conserving compute power. A tunable entity of many recent families of Intel x86 servers is the “uncore” component of the processor socket. The uncore functions are separate from actual processor cores including last level cache, memory and I/O control systems. This paper explores how detuning uncore frequency can modulate LLM performance by reducing it deterministically to within acceptable bounds. Since a corresponding power reduction is achievable, this is a very promising approach to achieving LLM energy efficiency at the network edge. As an example, for three concurrent llama3 instances, reducing uncore frequency from 2200 MHz to 1500 MHz drops the Tokens Per Second (TPS) from 6.78 to 6.45 with a corresponding 12 W power saving; further de-tuning to 1100 MHz yields a 32 W power saving while reducing TPS to 5.74, a value that is still marginally above a target “minimum acceptable” threshold.