MosaicKV: SLO-Aware KV-Cache Virtualization for Multi-Tenant LLM Serving in the Cloud
Abstract
Large language model (LLM) serving has become a first-class cloud workload. The dominant cost driver is no longer compute alone. The key-value (KV) cache consumes tens of gigabytes per replica. It forces providers to either over-provision GPU memory or reject long-context requests. Existing high-throughput engines largely optimize within a single server. They assume KV stays resident in GPU HBM. Under multi-tenancy, HBM is routinely oversubscribed. Naive offloading (e.g., CUDA Unified Memory paging) then creates severe p99 TTFT spikes. These spikes break service-level objectives (SLOs). This paper presents MosaicKV, a KV-cache virtualization layer for multi-tenant LLM inference on shared GPU servers. MosaicKV exposes KV as a schedulable and migratable resource. It jointly optimizes GPU-memory partitioning across tenants. It decides which KV pages stay on HBM versus spill to host DRAM. It also times prefetch and micro-batching to hide PCIe DMA transfer latency. The key insight is that decoding has a predictable layer-by-layer access pattern. By co-designing page layout with a slack-aware scheduler, MosaicKV turns KV offload from reactive paging into a planned pipeline. On an 8-GPU server (8 × H100 SXM 80 GB) serving Llama-3.1-70B, Qwen2.5-32B, and Mixtral-8x7B under a 64-tenant mixed workload, MosaicKV improves goodput-under-SLO by 1.93–2.31 × under the same SLO contract (p99 TTFT ≤ 400 ms for interactive tenants; p99 completion ≤ 2 s for best-effort tenants) while reducing peak HBM KV footprint by 38%. Under flash-crowd bursts, MosaicKV keeps interactive-tenant p99 TTFT at 378–386 ms (within SLO). Unified Memory spikes to 1,055 ms (∼ 2.6 × SLO). MosaicKV achieves these gains without model changes. Runtime overhead remains modest (2.9–6.2%). This enables operators to safely run long-context and bursty tenants on shared GPU fleets.