Skip to content
Log in
Open access

CELLServe: An SLO-Aware and Cost Efficient LLMs Serving System for Serverless Computing Environments

Zejian Wang Nan Lin Zinuo Cai Rongbo Ma Ruhui Ma Haibing Guan Rajkummar Buyya
Aug 2026 · ACM Transactions on Architecture and Code Optimization (TACO) · 0 citations · 1 references

TL;DR

CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.

Abstract

Large Language Models (LLMs) have enabled diverse AI applications. However, LLMs inference impose unprecedented computational and memory overhead, creating an inherent trade-off between latency Service Level Objectives (SLOs) and resource constraints. Serverless computing, with on-demand provisioning and pay-as-you-go billing, is becoming a promising paradigm for LLM serving. But existing solutions fail to integrate state-of-the-art inference optimizations, resulting in suboptimal GPU utilization and prolonged latency. While Prefill-Decode (PD) disaggregation combined with continuous batching has resolved such inefficiencies in traditional cloud deployments, migrating these techniques to serverless makes two challenges particularly pronounced: (1) SLO-constrained resource provisioning for independently scaling prefill and decode phase functions, and (2) function lifespan management to mitigate resource waste from continuous batching-induced prolonged instance lifespans. To tackle these issues, we propose CELLServe, an SLO-aware and cost-efficient serverless LLM serving system that pioneers integrating PD disaggregation and continuous batching into serverless platforms. CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources. Comprehensive evaluations on five mainstream LLMs and real-world traces show that CELLServe achieves 1.85×–1.92×higher request throughput than baselines under identical SLOs and GPU budgets, while sustaining high resource efficiency under dynamic workloads.

DOI Semantic Scholar
Read PDF

Similar papers

Preprint Aug 2026

OpScale: Operator-level Provisioning and Autoscaling for LLM Serving

OpScale is presented, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving that attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.

Xingqi Cui, Chieh-Jan Mike Liang, Ziang Tang et al. · 0 citations
Preprint Aug 2026

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

Cascade, an LLM serving system that estimates and continuously updates this per-request latency budget from request characteristics, KV-cache state, and current system load, and uses a single per-request budget to jointly coordinate request scheduling and KV-cache management across the memory hierarchy.

Muhammad Adnan, R. Mahapatra, Prashant J. Nair et al. · 0 citations
Book Open access Aug 2026

Towards Efficient Serving of Network-intensive LLM Inferences

Prefix caching has become a key technique for LLM serving, and nowadays the reusable KVCache contents are often hosted on distributed servers. For long-context LLM inferences with high cache hit ratio, cross-server KVCache transmission has become an emerging performance bottleneck; such network-intensive LLM inferences are increasingly prevalent in the coming era of agentic AI. However, existing LLM inference engines are essentially compute-centric; we find that they are highly inefficient when serving such workloads due to compute-stage service blocking and ignorance of KVCache-transfer cost. To efficiently serve network-intensive LLM inferences, in this paper, we design Sanic, an optimized LLM engine that treats KVCache transmission as a first-class citizen. Viewing KVCache loading and computation as equally-significant stages, Sanic decouples their service control and allows each stage to progress autonomously in an asynchronous manner, thereby improving the overall resource utilization. Moreover, when scheduling competing LLM inferences, Sanic treats the KVCache loading delay as an independent factor in service cost modeling, which is more accurate and can yield better scheduling decisions. Our testbed experiments with diverse benchmarks show that, Sanic can substantially enhance the service efficiency of network-intensive LLM inferences, improving the SLO-attainment by up to 61.67%.

Weiye Wang, Chen Chen, Junxue Zhang et al. · 0 citations
Jun 2026

Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs

Festina is presented, a profiling-guided, power-aware control plane to minimize cluster-wide energy for serverless LLM serving and performs energy-aware workload consolidation to reduce GPUs'static power consumption via SLO-aware migration.

Tianyu Wang, Gourav Rattihalli, A. Dhakal et al. · 0 citations