Co-Optimizing Request Scheduling and KV Caching for Edge LLM Serving

CELLServe: An SLO-Aware and Cost Efficient LLMs Serving System for Serverless Computing Environments

CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.

Janus: Joint Prefill/Decode Disaggregation with KV-Cache-Aware Multi-Cloud Routing for Edge-Adjacent LLM Serving

Janus is, to the authors' knowledge, the first scheduler to treat the KV-transport decision as a first-class scheduling variable jointly with prefill and decode routing across heterogeneous multi-cloud fleets, with provable guarantees.

Robust KV Cache Management for LLM Serving under Output Token Length Uncertainty

This work presents a robust KV cache management framework for LLM serving that jointly optimizes GPU parallelism configuration, KV cache reservation per request class, request routing across heterogeneous serving groups, and prefix caching for shared prompts that incorporates latency SLO constraints and captures the interaction between memory allocation, throughput, and queueing delay.

Towards Efficient Serving of Network-intensive LLM Inferences

Prefix caching has become a key technique for LLM serving, and nowadays the reusable KVCache contents are often hosted on distributed servers. For long-context LLM inferences with high cache hit ratio, cross-server KVCache transmission has become an emerging performance bottleneck; such network-intensive LLM inferences are increasingly prevalent in the coming era of agentic AI. However, existing LLM inference engines are essentially compute-centric; we find that they are highly inefficient when serving such workloads due to compute-stage service blocking and ignorance of KVCache-transfer cost. To efficiently serve network-intensive LLM inferences, in this paper, we design Sanic, an optimized LLM engine that treats KVCache transmission as a first-class citizen. Viewing KVCache loading and computation as equally-significant stages, Sanic decouples their service control and allows each stage to progress autonomously in an asynchronous manner, thereby improving the overall resource utilization. Moreover, when scheduling competing LLM inferences, Sanic treats the KVCache loading delay as an independent factor in service cost modeling, which is more accurate and can yield better scheduling decisions. Our testbed experiments with diverse benchmarks show that, Sanic can substantially enhance the service efficiency of network-intensive LLM inferences, improving the SLO-attainment by up to 61.67%.

Sparsity-Aware Scheduling for Efficient MoE-Based LLM Serving on Edge Servers

PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences

With rising popularity of LLMs, the performance, scalability, and resource-efficiency of inferences become a crucial challenge. The core part of the inference process is the KV cache, which avoids recomputing intermediate attention states, and the batching strategy that batches multiple requests per forward pass to leverage GPU parallelism. KV cache memory grows linearly with sequence length and batch sizes, easily exceeding the limited GPU memory capacity. State-of-the-art inference runtimes use continuous batching to maximize GPU utilization by interleaving the processing of new requests (i.e., prefill requests) with ongoing generation requests (i.e., decode requests). However, existing schedulers greedily admit prefill requests without considering the future KV cache memory required to successfully run the decode phases. This shortsighted approach causes frequent KV cache overflows, which in turn trigger preemption and recomputation of requests, severely degrading both throughput and latency. We propose PKAS, a Predictive KV Cache-Aware Scheduling algorithm to mitigate this inefficiency by reducing preemptions. PKAS uses a low-overhead technique to simulate future KV cache utilization and guide the admissibility for new request candidates. Combined with lightweight output-length predictions, PKAS can make better batching decisions, preventing KV cache overflows and drastically reducing preemptions. Evaluations on diverse models and workloads show that PKAS achieves up to 7.34x higher throughput and 8x lower latency compared to state-of-the-art scheduling, with the largest gains on long-context workloads where KV cache pressure is high.