MosaicKV: SLO-Aware KV-Cache Virtualization for Multi-Tenant LLM Serving in the Cloud
Large language model (LLM) serving has become a first-class cloud workload. The dominant cost driver is no longer compute alone. The key-value (KV) cache consumes tens of gigabytes per replica. It forces providers to either over-provision GPU memory or reject long-context requests. Existing high-throughput engines larg...