Jul 2026· IEEE International Conference on Cloud Computing· pp. 268-278· 0 citations· 27 references
Abstract
Large Language Model (LLM) inference services increasingly rely on heterogeneous GPU clusters to balance cost and performance. However, request routing in such environments is challenging because schedulers must account for hardware heterogeneity, dynamic workload characteristics, and bursty arrivals. Existing approaches either ignore hardware differences, rely on static heterogeneity-aware allocation, or use queue-based proxies that fail to capture the true remaining work under load. We present BYSTANDER, a prediction-based scheduling framework that uses a small language model (SLM) to estimate End-to-End (E2E) latency of request for each GPU pool from request features and current pool state. BYSTANDER then adaptively narrows the candidate pools via Fisher–Jenks grouping and performs queue-aware selection within that set. Its pool-based design keeps prediction overhead scalable as cluster size grows. We evaluate BYSTANDER on ShareGPT and LMSYS-Chat workloads over heterogeneous clusters of 3–7 GPUs (RTX 3090/4090/5090). Compared with round-robin, oracle-weighted round-robin, shortest-queue-first, and SLM Adaptive baselines, BYSTANDER reduces P99 E2E latency by up to 63.4% and P99 time-to-first-token by up to 87.4% under bursty traffic.
Load balancers in practice often rely on fixed heuristics such as weighted round-robin (WRR) or least connection (LC). Although these methods scale well, they do not capture differences in backend service capacity or runtime performance variations. which can increase tail latency and request drop rates in shared clusters. This issue becomes particularly significant for modern cloud inference workloads such as large language model (LLM) serving, where request processing times and backend utilization can vary substantially. To address this issue, we propose LP-WRR, an external control-plane mechanism that adjusts weighted round-robin weights using latency feedback. Our method periodically solves a constrained linear program (LP) to reduce expected service latency, then smooths the resulting weights with an exponential moving average so that updates do not fluctuate excessively, all while keeping the optimization logic off the data path. We then evaluate LP-WRR in a simulator for shared latency-sensitive services, using workloads derived from seven days of Azure ChatGPT traces as a representative inference workload, and compare it with the original load-balancing algorithm. Across the heterogeneity scenarios at low workload, our method improves both mean and tail latency, reducing p95 latency by $\mathbf{2 4. 7 \%}$ on average compared to baseline policies. Under the highest tested load, it also lowers the system-wide drop ratio in both homogeneous and heterogeneous infrastructure.
Hai Pham Thanh, Dang Hoang Nguyen, Anh Nguyen Tuan et al.· IEEE International Conferenc...· 0 citations
The widespread adoption and strong generalizability of large language models (LLMs) lead to highly heterogeneous workloads that exhibit substantial variability in request lengths and latency requirements. This pronounced heterogeneity causes existing scheduling strategies to suffer from head-of-line blocking and inefficient request ordering, resulting in significant SLO violations and inflated end-to-end latency. In this paper, we present UNAS, a hierarchical scheduler for end-to-end SLO-oriented LLM serving. UNAS adopts a two-level design. At the cluster level, it classifies incoming requests into urgent and normal queues based on estimated service times derived from request features and runtime statistics, and dynamically updates their queue priorities as latency pressure evolves. At the instance level, UNAS performs fine-grained dispatch using a least-slack-first policy, prioritizing requests with the highest risk of SLO violations. Experimental results demonstrate that UNAS consistently outperforms representative baselines across diverse workloads, improving end-to-end SLO attainment by up to 11.1% and reducing mean end-to-end latency by up to $1.47 \times$.
Tian-Nan Fu, Jianxiong Liao, Xu Chen et al.· Fall Joint Computer Conferen...· 0 citations
Cascade, an LLM serving system that estimates and continuously updates this per-request latency budget from request characteristics, KV-cache state, and current system load, and uses a single per-request budget to jointly coordinate request scheduling and KV-cache management across the memory hierarchy.
Muhammad Adnan, R. Mahapatra, Prashant J. Nair et al.· 0 citations
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Zejian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
Existing Large Language Model (LLM) inference systems often rely on static model placement and scheduling policies, which struggle to handle heterogeneous and dynamic real-world workloads. The key challenge is to adapt serving strategies to workload fluctuations while keeping reconfiguration overhead minimal. In this paper, we present OrionInfer, an adaptive LLM serving system that aligns inference strategies with real-time demand. OrionInfer introduces three key techniques: (1) runtime switching between data parallelism and tensor parallelism with negligible overhead; (2) an efficient inference pipeline that preserves batching efficiency during parallelism transitions; and (3) live-migration-based load balancing to alleviate memory pressure and improve resource utilization. Evaluations across multiple model scales show that OrionInfer delivers robust performance under diverse serving scenarios. In end-to-end serving, it reduces average TTFT by up to 25% over DP-priority configurations under low loads and lowers P99 tail latency by 50%--90% over TP-priority configurations under most high-traffic settings. In disaggregated prefill serving, OrionInfer improves prefill completion time (PCT) SLO attainment by up to 16.5 percentage points over DP-priority static baselines and reduces P99 PCT by up to 74.7% over TP-priority static baselines. Compared with dynamic baseline, OrionInfer provides better tail-latency stability, reducing P99 PCT by 38.6%--40.8% while avoiding the extra memory footprint.
Jingqi Feng, Guang Yang, Yukai Huang et al.· Proceedings of the 32nd ACM...· 0 citations
DeltaServe is presented, a host-agnostic co-serving design that converts this idle inference capacity into LoRA fine-tuning throughput while preserving inference service-level objectives (SLOs).
Jiaxuan Chen, Jianshu She, Ye Yuan et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
