Open access

AdaptiveKV: Accelerating KV Cache Offloading with a Bandwidth-Adaptive Memory Allocation Mechanism

Yibo Tang Lizhou Wu Yang Ou Sunfeng Gao Yanjing Wang Zicong Wang Xingyun Qi Jiaqing Xu Fangxu Lv Liquan Xiao Mingche Lai

Jul 2026 · ACM Transactions on Architecture and Code Optimization (TACO) · Vol 23, pp. 1-25 · 0 citations · 54 references

TL;DR

A bandwidth-oriented memory allocation mechanism, named AdaptiveKV, which is self-adaptive to CXL-enabled memory pools and KV cache offloading scales for LLM inference acceleration, and achieves a maximum speedup in LLM inference throughput compared to the state-of-the-art strategies.

Abstract

The explosive growth of key-value (KV) cache size in large language model (LLM) inference poses a key challenge to the limited HBM of GPU. Offloading KV cache to host memory has become a prevalent mitigation method. However, the limited host DDR bandwidth, especially in multi-GPU inference scenarios, often leads to offloading bottlenecks, thereby restricting inference speed. Compute express link (CXL) offers a promising alternative to expand host memory capacity and bandwidth on demand. In this paper, we present a bandwidth-oriented memory allocation mechanism, named AdaptiveKV, which is self-adaptive to CXL-enabled memory pools and KV cache offloading scales for LLM inference acceleration. Our systematic profiling of CXL-HBM memory bandwidth under GPU workloads reveals that conventional memory strategies neglect dynamic memory bandwidth fluctuations and various CXL memory characteristics, leading to suboptimal memory utilization. Motivated by these insights, AdaptiveKV implements three core designs: 1) a GPU memory conch model to guide memory allocation strategies, 2) a runtime predictor to predict optimal memory allocation ratios, and 3) a dynamic interleaving strategy to allocate memory pages across available NUMA nodes. Experimental results suggest that AdaptiveKV achieves a maximum speedup of 1.90 × in LLM inference throughput compared to the state-of-the-art strategies. To further explore AdaptiveKV’s applicability boundary, we also present an FPGA-based CXL memory emulator with configurable performance, revealing that a CXL-to-DDR bandwidth ratio exceeding 8% yields at least a 5% speedup in LLM inference.

DOI Semantic Scholar

Read PDF

Similar papers

Jul 2026

Elastic Memory Remapping for Multi-tenant LLM Serving

This work presents Oneiros, a dynamic remapping engine for multi-tenant LLM serving that dynamically repurposes GPU memory allocated for model parameters as KV cache capacity, enabling nonblocking, unidirectional parameter transfer.

Ruihao Li, Shagnik Pal, Vineeth Narayan Pullu et al. · ACM SIGOPS Operating Systems... · 0 citations

Preprint Aug 2026

OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

OasisKV is presented, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding and observes that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD).

Can Xiao, Sukmin Cho, J. We et al. · 0 citations

Preprint Jul 2026

FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference

FlashAccel integrates HBF into HBM-based GPUs, providing architectural support to mitigate access latency and introduces an HBF-aware storage management layer together with a programming model to organize persistent data in HBF and coordinate heterogeneous memory resources at the system level.

Xinyu Wang, Yalong Xue, Xiaotian Sun et al. · 1 citation

Understanding and Optimizing KV-cache Management for Long-Context LLM Inference A

This model reveals one key opportunity: dividing a restore request proportionally between the storage path and the GPU can improve inference performance while still meeting SLOs, and reduces the KV-cache storage stack to a performance model based on per-tier capacity, per-tier and interconnect bandwidth, and GPU arithmetic throughput to identify optimization opportunities for KV-cache management.

Unknown authors · 0 citations

Jul 2026

A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference

LLM inference at scale faces a memory wall. The KV cache demands tens of terabytes at hundreds of gigabytes per second, yet no current memory tier delivers both at once. Characterization across multi-generation GPU systems with various LLaMA models shows host memory retrieval achieves up to 100x speedup over re-computation but supports only tens of concurrent long-context users. Electrical CXL pooling theoretically bridges this gap, but switch latency, cable reach limits, and power-scaling issues prevent practical TB-scale deployments. We present the Marvell Photonic Fabric Memory Appliance, a photonic-CXL hybrid architecture replacing electrical switches with a passive fiber shuffle to deliver 32 TB shared memory across 16 hosts via a switch-free full- crossbar topology. Emulation results demonstrate over 50 percent latency reduction versus electrical CXL pools. Simulation results show that the PF Memory Appliance eliminates cache eviction cliffs by improving time-to-first-token by 6.6x for multi-turn conversations workloads.

Jing Ding, Yash Nishant, Chandrish Ambati et al. · arXiv.org · 0 citations

Book Open access Jul 2026

Scaling Attention Beyond GPUs for LLM Inference

Beyond is presented, a drop-in runtime that integrates a smart offloading scheme to selectively identify and retain salient KV entries across continuous decoding sessions, together with a hybrid CPU–GPU attention mechanism for scalable inference.

Weishu Deng, Yujie Yang, Peiran Du et al. · IEEE International Symposium... · 1 citation