Sep 2026· Proceedings of the International Conference on Parallel Processing· 0 citations· 8 references
TL;DR
WAQ-LLM is proposed, a performance optimization framework to find the optimal deployment configuration for multi-instance LLM serving that can provide adaptive and hybrid configurations to handle diverse workloads, surpassing static deployments limited to a single instance type.
Abstract
The deployment of Large Language Models (LLMs) on multi-instance GPU clusters has become essential to meet the surging demand for generative AI applications. While scaling out instances increases throughput, the distinct computational characteristics of prefill and decode phases introduce significant resource contention, making it challenging to satisfy stringent Service Level Objectives (SLOs) for responsiveness and generation speed. Existing serving solutions typically rely on static deployment strategies—either aggregating or disaggregating these phases—which often fail to adapt to the interference patterns caused by varying request arrival rates and sequence lengths. In this work, we propose WAQ-LLM, a performance optimization framework to find the optimal deployment configuration for multi-instance LLM serving. Specifically, we first establish an analytical workload-aware queueing model that captures the LLM computational characteristics and queuing behavior of both aggregation and disaggregation designs. We then formulate the deployment configuration problem as a constrained optimization problem and develop a polynomial-time algorithm to efficiently identify configurations that minimize Time-per-Output-Token (TPOT) while satisfying SLOs. Extensive experiments are conducted on a 16-GPU cluster across various LLMs and workload settings. WAQ-LLM consistently outperforms AIConfigurator, NVIDIA’s official product-level solution, reducing TPOT by 21.62% on average and up to 81.7%. Moreover, WAQ-LLM can provide adaptive and hybrid configurations to handle diverse workloads, surpassing static deployments limited to a single instance type (either PD aggregation or disaggregation).
Janus is a Service-Engine co-designed multi-model serving system built around dual-timescale scheduling that maintains 0.97-1.0 SLO attainment, versus 0.80-0.92 for the strongest baseline, while using 13,440 device-hours/day, 27% less than static Serverless-LLM.
Tian-Bao Zhou, Yi Wang, Yu Zhou et al.· Proceedings of the ACM SIGOP...· 2 citations
CrossServe is introduced, a cross-layer scheduling framework that jointly optimizes request routing, adaptive micro-batching, and tenant-weighted preemption to maximize SLO attainment in multi-tenant LLM serving and increases overall SLO attainment by 53, 29, and 14 percentage points respectively.
Jia-He Li, Jia-Bin Li· Proceedings of the Internati...· 0 citations
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Ze-Jian Wang, Nan Lin, Zi-Nuo Cai et al.· ACM Transactions on Architec...· 0 citations
OpScale is presented, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving that attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.
Xingqi Cui, Chieh-Jan Mike Liang, Ziang T. Tang et al.· 0 citations
Recent advances in large language models (LLMs) are driving the emergence of multi-modal and agentic services for mobile users through cloud and edge infrastructures, where long-context workloads pose daunting challenges for inference latency. Existing disaggregated LLM serving systems largely rely on hardware profilin...
Large Language Models (LLMs) are increasingly deployed in latency-sensitive applications, where real-time serving must satisfy stringent service-level objectives (SLOs). However, request intensities fluctuate over time, and under low load LLM services leave a substantial fraction of GPU task idle. A promising approach...
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 12, 2026
MIT scientists developed a training method to generate models that simulate the effects of human activities on climate much more quickly than other approaches.
Microsoft Research Blog· microsoft.comNov 10, 2026
Microsoft researchers have discovered “Lazarus,” a backdoor technique that can reprogram dormant neurons in an AI model, causing an otherwise normal AI agent to execute malicious commands when a hidden trigger appears. The threat is especially serious for AI agents with tool or terminal access. A triggered model could quietly download and run attacker code, […] The post Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents appeared first on Microsoft Research.