GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference

Abstract

Diffusion large language models (dLLMs) are emerging as a promising generative paradigm that complements autoregressive decoding. In long-context settings, KV cache bloat and offloading transfer overhead have become primary bottlenecks in inference systems. Meanwhile, the periodic full-sequence recomputation and localized token updates in dLLMs make the KV lifecycle substantially more dynamic, complicating cache management and prefetch scheduling while making heavyweight token-level indexing or clustering schemes harder to amortize effectively during decoding. To address these challenges, we present GroupKV, a lightweight hierarchical KV cache management system for long-context dLLM inference. We observe that under block-wise decoding, tokens within the same generation block tend to access highly overlapping and spatially concentrated context regions, making group-level sparse selection effective. Building on this observation, GroupKV partitions the context into contiguous groups and performs coarse-to-fine sparse selection. GroupKV further exploits cross-layer consistency to enable predictive prefetching, and incorporates a staleness correction mechanism to maintain cache coherence under dynamic KV updates. Additionally, GroupKV adopts streaming prefill to reduce peak memory consumption during prefilling. Experiments show that GroupKV extends the maximum serviceable context length by up to 48.00 × under constrained GPU memory, improves end-to-end inference performance by up to 3.73 × in offload-based long-context settings, and maintains competitive task accuracy.