Jul 2026· IEEE International Conference on Cloud Computing· pp. 388-394· 0 citations· 40 references
Abstract
Self-hosting multiple large language models on private or hybrid cloud infrastructure is increasingly appealing for organizations seeking privacy, cost control, and customization, yet no existing system coordinates model selection with model lifecycle management. Serving frameworks such as KServe and Ray Serve deploy models but lack query-aware routing. Routing frameworks such as RouteLLM and FrugalGPT select models but ignore whether those models are warm or cold. We present Pick and Spin, a multi-model orchestration framework built on Kubernetes that unifies these concerns. Pick classifies query complexity using a hybrid keyword-DistilBERT approach and selects models via Thompson Sampling, incorporating lifecycle state from Spin. Spin tracks per-model warm and cold states through a Cold Start State Machine, factors I/O-bound weight-loading penalties into routing decisions, and manages warm pools for frequently accessed models. The two components form a closed feedback loop where routing decisions account for resource availability and orchestration anticipates routing patterns. Deployed as a Helm umbrella chart serving nine LLMs (1B–27B parameters) via vLLM on NVIDIA A100 GPUs, Pick and Spin achieves 61% GPU utilization versus 31% for static deployment, reduces GPU-hours by 46%, and maintains a 9.2% per-query cold start rate across 310,190 total inference runs-31,019 queries evaluated against all nine models for ground truth, then routed through Pick and Spin-spanning eight standard benchmarks.
CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance targets through per-pair independent scoring, eliminating multi-positive dilution while regularizing queries with similar routing preferences to be closer in the routing space.
LLM-Advisor is introduced, an open-source, adaptive framework designed for intelligent query categorization, dynamic model evaluation, and constraint-aware request routing across heterogeneous multi-LLM pools that achieves a 42% reduction in overall inference expenditure and a 35% decrease in average response latency.
Harshil Lodhiya· International Journal of Res...· 0 citations
A Task-Oriented Prefix-Aware Scheduler that jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution and scores candidate post-decision states by trading off the expected reduction in each task's longest remaining service path against the near-term benefit of downstream prefix reuse.
Hongqiu Ni, Han Tian, Chi Zhang et al.· 1 citation
Deploying Large Language Models (LLMs) over the edge-cloud continuum faces severe stability challenges due to the conflict between stochastic network topology and complex workflow dependencies. Existing schedulers, relying either on computationally prohibitive Graph Neural Networks (GNNs) or topology-agnostic heuristics, fail to reconcile this tension. To bridge these gaps, we propose STEM, a service-level and topology-aware orchestration framework that formulates distributed LLM serving as a workflow-aware routing problem over a monitored service overlay, in which heterogeneous service instances act as specialized experts. At the core of STEM lies the STAR-PPO algorithm, utilizing a lightweight graph-free perception mechanism. By leveraging Squeeze-and-Excitation attention, it extracts critical bottleneck features from raw telemetry with linear complexity, bypassing the scalability limits of message-passing paradigms. To further achieve Pareto-efficient trade-offs, we develop a Dynamic Weight Adaptation (DWA) mechanism that autonomously recalibrates optimization preferences based on entropy-regularized metric drift. Extensive experiments on real-world datasets spanning 2,000 nodes demonstrate that our framework significantly outperforms state-of-the-art baselines. Specifically, STAR-PPO reduces network transmission costs by 96.8% and improves comprehensive inference efficiency by 24.4%, while sustaining robust zero-shot generalization across regions, with average latency within $1.09\times $ of a target-domain-retrained reference under a strict cross-region protocol. Code and data are available at https://github.com/gymorsiback/STARPPO
Yan Gao, Shaoyuan Huang, Yonghui Ye et al.· IEEE Transactions on Cogniti...· 0 citations
Large Language Model-based Multi-Agent Systems (LLM-MAS) have shown exceptional promise for complex tasks, including retrieval-augmented generation and autonomous data analytics. However, their deployment in resource-constrained industrial environments faces critical challenges, such as unpredictable end-to-end latency, low task success rates, and expensive operational costs. Existing orchestration strategies for LLM-MAS mainly rely on static heuristics, implicit LLM-based routing, or reinforcement learning, which suffer from brittleness under workload drift, prohibitive online exploration costs, and inherent governance risks. To address these issues, we propose G-STAR, a general graph-based scheduling framework that formalizes complex MAS pipelines as attributed Directed Acyclic Graphs (DAGs). Specifically, G-STAR decouples its learning pipeline into three critical stages to ensure both system stability and execution efficiency. First, we build a data layer that logs fine-grained signals for agent node (e.g., execution latency, token I/O, model IDs, resource metrics) and task-grounded outcomes (e.g., accuracy, cost, SLA), yielding a trace-driven dataset. Second, we train an offline GNN model to predict optimal dynamic routing decisions, including agent activation, concurrency-constrained Top-K selection, and execution ordering, conditioned on the evolving graph state. Finally, a lightweight dispatcher deploys the pre-trained GNN model for single-forward-pass routing with negligible computational overhead. Extensive experiments on two public benchmarks and our real-world production workload confirm G-STAR's superiority over static and LLM-driven baselines. It maintains competitive task quality while reducing latency variability and consistently improving end-to-end latency. Furthermore, we develop an industry-grade orchestration stack with asynchronous execution, resilient serving, and audit-friendly artifacts, offering a practical solution for optimizing web-scale deployments of complex MAS pipelines.
Jiabao Song, Yunsheng Xia, Beibei Kong et al.· Proceedings of the 32nd ACM...· 0 citations
Multi-agent applications increasingly rely on shared large language model backends in the public cloud, where bursty workloads cause requests from different agents to contend for the same LLM instances, leading to long queues, memory imbalance, and severe tail-latency inflation. Existing approaches typically prioritize requests using coarse workflow positions or static execution heuristics, which fail to adapt to short-term overload dynamics. We present FlowGuard, a workflow-aware overload controller for multi-agent LLM serving. Its key insight is that under sustained overload, GPU cycles spent on requests whose execution service-level-objectives (SLOs) are already violated are wasted. FlowGuard continuously recomputes per-request slack and prioritizes requests with the greatest remaining time before their deadlines, thereby maximizing on-time completions. In addition, a resource-aware dispatcher jointly accounts for KV-cache memory pressure and in-flight queue depth to reduce preemption across shared instances. Evaluated on a deliberately over-subscribed two-GPU backend, where all policies exhibit high absolute miss rates (i.e., the percentage of workflows that miss their deadlines), FlowGuard reduces the miss rate by 14–28% points over workflow-oblivious and static-priority baselines under BurstGPT-driven load, and by 34–38% points under co-located mixed-agent workloads.
Ali Zafar Sadiq, Hai-Ying Shen· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
