Skip to content
Book Open access

MosaicKV: SLO-Aware KV-Cache Virtualization for Multi-Tenant LLM Serving in the Cloud

Nov 2026 · Proceedings of the 2026 ACM Symposium on Cloud Computing Volume 1 · 0 citations · 42 references

Abstract

Large language model (LLM) serving has become a first-class cloud workload. The dominant cost driver is no longer compute alone. The key-value (KV) cache consumes tens of gigabytes per replica. It forces providers to either over-provision GPU memory or reject long-context requests. Existing high-throughput engines largely optimize within a single server. They assume KV stays resident in GPU HBM. Under multi-tenancy, HBM is routinely oversubscribed. Naive offloading (e.g., CUDA Unified Memory paging) then creates severe p99 TTFT spikes. These spikes break service-level objectives (SLOs). This paper presents MosaicKV, a KV-cache virtualization layer for multi-tenant LLM inference on shared GPU servers. MosaicKV exposes KV as a schedulable and migratable resource. It jointly optimizes GPU-memory partitioning across tenants. It decides which KV pages stay on HBM versus spill to host DRAM. It also times prefetch and micro-batching to hide PCIe DMA transfer latency. The key insight is that decoding has a predictable layer-by-layer access pattern. By co-designing page layout with a slack-aware scheduler, MosaicKV turns KV offload from reactive paging into a planned pipeline. On an 8-GPU server (8 × H100 SXM 80 GB) serving Llama-3.1-70B, Qwen2.5-32B, and Mixtral-8x7B under a 64-tenant mixed workload, MosaicKV improves goodput-under-SLO by 1.93–2.31 × under the same SLO contract (p99 TTFT ≤ 400 ms for interactive tenants; p99 completion ≤ 2 s for best-effort tenants) while reducing peak HBM KV footprint by 38%. Under flash-crowd bursts, MosaicKV keeps interactive-tenant p99 TTFT at 378–386 ms (within SLO). Unified Memory spikes to 1,055 ms (∼ 2.6 × SLO). MosaicKV achieves these gains without model changes. Runtime overhead remains modest (2.9–6.2%). This enables operators to safely run long-context and bursty tenants on shared GPU fleets.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.