Skip to content

Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving

Sep 2026 · 0 citations · 31 references
Computer Science Engineering

TL;DR

Results demonstrate that learned routing can selectively allocate expensive model capacity based on query context and improve the accuracy-energy tradeoff in multi-LLM serving.

Abstract

Large language models (LLMs) and agentic AI systems are creating rapidly growing inference energy demands as model sizes grow and reasoning trajectories extend. While in practice, many queries do not require the capabilities of the largest available model, and routinely directing such queries to a high-capability model can introduce unnecessary, considerable computation and energy consumption. In this paper, we investigate whether adaptive routing across a heterogeneous pool of LLMs can reduce this energy burden without substantially compromising task performance. We design a language-model-based router that reads in each query and selects an answer model from a fixed candidate pool. The candidate models are first profiled through an offline tournament that records their correctness, latency, power, and GPU energy for each query. Using these measurements, the router is trained through supervised fine-tuning followed by group relative policy optimization (GRPO) with the tailored paradigms. Results demonstrate that learned routing can selectively allocate expensive model capacity based on query context and improve the accuracy-energy tradeoff in multi-LLM serving. Across seven benchmark tasks, we also observe a sharp accuracy-energy phase transition among routers, providing practical insights into improving energy efficiency while maintaining LLM performance.

View source

Similar papers

Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing

This work introduces RouteFM, which learns to characterize anonymous candidate models from behavioral context and infer their target-specific capabilities, rather than binding routing decisions to fixed model identities or a single environment, and shows transfer across changes in domains, modalities, candidate pools,...

Guannan Lai, Han-Jia Ye · 1 citation

ARM: Attention with Routed-Memory for Learnable Sparse Control

Attention with Routed Memory is proposed a novel KV caching structure that introduces a fully differentiable, fixed-size memory system organized as a hierarchical router that achieves superior performance and efficiency compared to fixed KV-caching approaches, while remaining efficient and scalable in terms of both mem...

Qiu-Hao Zeng, Jerry M. Huang, Peng Lu et al. · 0 citations
Preprint Aug 2026

RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning

RLCascadeRouter is a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising ``stop''and model selection, and uses trajectory returns and advantages to directly optimize the performance-cost objective.

Shihong Huang, Sheng-Jie Wang, Hong-Yao Ma et al. · 1 citation
#machine learning Preprint Oct 2026

Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads

LLM deployment is shifting from single-turn completion to agentic trajectories in which a model plans, calls tools, reads results and reasons at test time before acting. This inverts the economics of inference hardware: chat serving amortises weight reads across large batches, whereas agent trajectories are sequentiall...

Abbas Ali, Muhammad Ajmal Siddiqui, Moona Zahid · 0 citations
#artificial intelligence Preprint Oct 2026

Memory-Efficient Expert Routing for Distributed MoE Training

As Mixture-of-Experts (MoE) models scale toward hundreds of experts and higher top-$k$ routing, memory efficiency in distributed training becomes a critical bottleneck. Peak memory is dominated by the MoE block, not attention: every intermediate buffer in the MoE dispatch pipeline is individually scaled by top-k routin...

Arnab Kanti Tarafder, Jaume Guasch-Martí, Gokcen Kestor et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.