LODAR: layered optimization for distributed adaptive LLM routing

Abstract

Large language model (LLM) serving platforms increasingly route requests across heterogeneous backends, transforming query routing from a model-selection task into a system-level load-balancing problem. Centralized routers can become throughput bottlenecks and single points of failure as deployment scale and request volume grow. Distributed routing mitigates these limitations, but it introduces a coupled min–max optimization problem in which multiple routers jointly determine peak backend utilization. This coupling makes routing optimization challenging under dynamic workload conditions. We propose LODAR, a layered optimization framework for distributed LLM routing. LODAR formulates routing as a distributed min–max load-balancing problem and optimizes routing logits directly through robust offline evolutionary search, router-level block-coordinate refinement, and lightweight online adaptation. This design does not require gradient-based credit assignment, yet exploits router-level structure and warm-start information across workload windows. Experiments on synthetic and WildChat-derived trace workloads cover three deployment scales, complemented by evaluation on a real heterogeneous multi-GPU testbed. In the scale-aggregated main comparisons, LODAR achieves the lowest MaxLoad and LoadStd under both workload settings. In trace-driven simulation, LODAR reduces MaxLoad by 16.9% and LoadStd by 40.3% relative to DE, the strongest evolutionary baseline in the simulation comparison. The simulation-optimized LODAR base routing table transfers directly to the physical deployment without deployment-specific offline retraining or tuning. In the real multi-GPU experiments, LODAR reduces MaxLoad by 4.7% and LoadStd by 14.9% relative to CMA-ES. These results demonstrate the effectiveness of LODAR’s layered optimization strategy for distributed load balancing in heterogeneous LLM serving systems.