EcoxPU: Autonomous Multi-Agent Orchestration for Energy-Efficient Disaggregated LLM Inference on Heterogeneous xPUs

Abstract

Large Language Model (LLM) inference is increasingly served on disaggregated clusters that combine different accelerator types, including NVIDIA GPUs, Huawei Ascend NPUs, and Kunlunxin AI processors. We show that these heterogeneous xPUs exhibit distinct energy-performance behaviors across inference phases and model modules, so a fixed deployment can waste substantial energy even when latency targets are met. This paper presents <monospace><bold>EcoxPU</bold></monospace>, an autonomous multi-agent framework for energy-efficient disaggregated LLM inference. EcoxPU jointly coordinates Prefill-Decode (PD) separation and Attention-Feed-Forward Network (FFN) disaggregation, assigning compute-bound and memory-bound work to suitable devices while adapting frequency and load placement at runtime. Evaluated on a heterogeneous cluster with NVIDIA H100, Huawei Ascend 910B, and Kunlunxin P800, EcoxPU reduces energy consumption by 31.9–45% (42% on average), while maintaining <inline-formula><tex-math notation="LaTeX">$\le$</tex-math><alternatives><mml:math><mml:mo>≤</mml:mo></mml:math><inline-graphic xlink:href="dong-ieq1-3732699.gif"/></alternatives></inline-formula>6% latency overhead and <inline-formula><tex-math notation="LaTeX">$\le$</tex-math><alternatives><mml:math><mml:mo>≤</mml:mo></mml:math><inline-graphic xlink:href="dong-ieq2-3732699.gif"/></alternatives></inline-formula>2% perplexity variation. It also improves MoE-FFN utilization on Ascend 910B to 82.3%, and cross-domain transfer learning accelerates agent adaptation by 2.67×.