Jul 2026· IEEE International Conference on Consumer Electronics· pp. 125-130· 0 citations· 16 references
Abstract
Load balancers in practice often rely on fixed heuristics such as weighted round-robin (WRR) or least connection (LC). Although these methods scale well, they do not capture differences in backend service capacity or runtime performance variations. which can increase tail latency and request drop rates in shared clusters. This issue becomes particularly significant for modern cloud inference workloads such as large language model (LLM) serving, where request processing times and backend utilization can vary substantially. To address this issue, we propose LP-WRR, an external control-plane mechanism that adjusts weighted round-robin weights using latency feedback. Our method periodically solves a constrained linear program (LP) to reduce expected service latency, then smooths the resulting weights with an exponential moving average so that updates do not fluctuate excessively, all while keeping the optimization logic off the data path. We then evaluate LP-WRR in a simulator for shared latency-sensitive services, using workloads derived from seven days of Azure ChatGPT traces as a representative inference workload, and compare it with the original load-balancing algorithm. Across the heterogeneity scenarios at low workload, our method improves both mean and tail latency, reducing p95 latency by $\mathbf{2 4. 7 \%}$ on average compared to baseline policies. Under the highest tested load, it also lowers the system-wide drop ratio in both homogeneous and heterogeneous infrastructure.
Existing Large Language Model (LLM) inference systems often rely on static model placement and scheduling policies, which struggle to handle heterogeneous and dynamic real-world workloads. The key challenge is to adapt serving strategies to workload fluctuations while keeping reconfiguration overhead minimal. In this paper, we present OrionInfer, an adaptive LLM serving system that aligns inference strategies with real-time demand. OrionInfer introduces three key techniques: (1) runtime switching between data parallelism and tensor parallelism with negligible overhead; (2) an efficient inference pipeline that preserves batching efficiency during parallelism transitions; and (3) live-migration-based load balancing to alleviate memory pressure and improve resource utilization. Evaluations across multiple model scales show that OrionInfer delivers robust performance under diverse serving scenarios. In end-to-end serving, it reduces average TTFT by up to 25% over DP-priority configurations under low loads and lowers P99 tail latency by 50%--90% over TP-priority configurations under most high-traffic settings. In disaggregated prefill serving, OrionInfer improves prefill completion time (PCT) SLO attainment by up to 16.5 percentage points over DP-priority static baselines and reduces P99 PCT by up to 74.7% over TP-priority static baselines. Compared with dynamic baseline, OrionInfer provides better tail-latency stability, reducing P99 PCT by 38.6%--40.8% while avoiding the extra memory footprint.
Jingqi Feng, Guang Yang, Yukai Huang et al.· Proceedings of the 32nd ACM...· 0 citations
Load Balancing has emerged as a critical problem in expert-parallel distributed inference of Mixture-of-Experts (MoE) models. As routing distributions are typically skewed across experts, devices hosting lighter-loaded experts must idle to wait for the heaviest during expert computing, leading to inefficiency. Existing load-balancing approaches primarily rely on expert replication or migration within each layer, which introduce additional overhead and limit their flexibility and scalability. To address this problem, we propose EasyBalance, a cross-layer load balancing strategy that requires no modifications to the expert-device mapping, enabling instant adaptability and incurring essentially no additional overhead. Our key insights are that (1) experts of other layers can be viewed as naturally redundant for the current layer, and (2) cross-layer MoE workloads can be jointly executed to mitigate their individual imbalance. Based on these observations, EasyBalance greedily schedules a subset of cross-layer workloads to run at each MoE step and defers the remaining workloads for future balancing opportunities, effectively leveraging cross-layer imbalance mitigation. Extensive experiments across models, tasks, and configurations demonstrate that EasyBalance consistently accelerates distributed MoE inference, reducing GPU idling by mostly over 40%. Code is available at https://github.com/yize-wu/EasyInfra.
The widespread adoption and strong generalizability of large language models (LLMs) lead to highly heterogeneous workloads that exhibit substantial variability in request lengths and latency requirements. This pronounced heterogeneity causes existing scheduling strategies to suffer from head-of-line blocking and inefficient request ordering, resulting in significant SLO violations and inflated end-to-end latency. In this paper, we present UNAS, a hierarchical scheduler for end-to-end SLO-oriented LLM serving. UNAS adopts a two-level design. At the cluster level, it classifies incoming requests into urgent and normal queues based on estimated service times derived from request features and runtime statistics, and dynamically updates their queue priorities as latency pressure evolves. At the instance level, UNAS performs fine-grained dispatch using a least-slack-first policy, prioritizing requests with the highest risk of SLO violations. Experimental results demonstrate that UNAS consistently outperforms representative baselines across diverse workloads, improving end-to-end SLO attainment by up to 11.1% and reducing mean end-to-end latency by up to $1.47 \times$.
Tian-Nan Fu, Jianxiong Liao, Xu Chen et al.· Fall Joint Computer Conferen...· 0 citations
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Zejian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
A lightweight latency estimator is designed that simulates autoregressive token batch processing in the serving framework and estimates the time-to-first-token (TTFT) of queries and is incorporated into a latency-aware router that jointly optimizes latency, accuracy, and cost when assigning queries to model instances.
Shivam Patel, Akaash R. Parthasarathy, Ankur Mallick et al.· arXiv.org· 0 citations
The method lets an HPC site compare deployment strategies using both user-visible performance and scheduler impact, then select the fastest strategy within its own RPC-demand limit.
Nil Tianchen Mu, William Dizon, Glen Otero et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
