Elastic Memory Remapping for Multi-tenant LLM Serving
TL;DR
This work presents Oneiros, a dynamic remapping engine for multi-tenant LLM serving that dynamically repurposes GPU memory allocated for model parameters as KV cache capacity, enabling nonblocking, unidirectional parameter transfer.
Abstract
KV cache accelerates LLM inference by avoiding redundant computation, but its rapidly growing memory footprint makes GPU memory a primary bottleneck in modern serving systems. Recent approaches extend GPU memory using CPU memory through KV-cache swapping. However, because KV cache is continuously updated during decoding, swapping introduces substantial synchronization and bidirectional transfer overheads. We present Oneiros, a dynamic remapping engine for multi-tenant LLM serving. Oneiros is based on a simple observation: unlike KV cache, model parameters remain immutable during inference. Instead of swapping KV cache itself, Oneiros dynamically repurposes GPU memory allocated for model parameters as KV cache capacity, enabling nonblocking, unidirectional parameter transfer. This approach is particularly effective in multi-tenant environments, where memory allocated to inactive models can be reclaimed dynamically for active workloads. We implement Oneiros in vLLM and evaluate it on modern GH200 systems. Compared to vLLM, Oneiros reduces tail latency by up to 99.3% and improves throughput by up to 86.7%. Compared to KV-cache swapping approaches, Oneiros achieves substantially higher throughput by avoiding synchronization overheads during runtime memory extension. Source code of Oneiros is available at https://github.com/UT-SysML/Oneiros/1.