DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch

Zuning Liang Zhiyi Yao Qi Chen Yuedong Xu Hao Dai Zhiqiang Ding Tongkai Yang Jinlong Hou Yuan Cheng

Jul 2026 · arXiv.org · Vol abs/2607.26475 · 1 citation

Computer Science

TL;DR

DualDecoder is presented, a lightweight serving system for long-context LLM inference that enables efficient sparse KV cache retrieval from host memory that leverages a novel dual-token decoding pipeline that accurately identifies critical KV entries with negligible computational overhead.

Abstract

Long-context inference is becoming a fundamental capability for modern LLM serving, especially driven by emerging agentic applications. Yet it faces a severe memory wall that the KV cache scales proportionally with increasing context length and request concurrency. Existing sparse KV cache methods offload most KV entries to host memory and retrieve only the critical KV entries needed by each decoding step. However, they commonly introduce substantial auxiliary states in GPU memory for KV retrieval management. Our measurements show that these often-overlooked auxiliary states introduce significant memory overhead and become a new bottleneck under high-concurrency workloads. In this paper, we present DualDecoder, a lightweight serving system for long-context LLM inference that enables efficient sparse KV cache retrieval from host memory. Our key insight is that the critical KV entries required for decoding the next token can be accurately predicted from the preceding speculated token. This predictability enables KV retrieval to be proactively prefetched and overlapped with decoding computation, effectively eliminating the GPU memory overhead of auxiliary states. To achieve this prefetching efficiently, DualDecoder leverages a novel dual-token decoding pipeline that accurately identifies critical KV entries with negligible computational overhead, and designs a layer-aware transfer schedule to overlap KV prefetching with model computation and a layer-scoped memory manager to reduce the GPU runtime buffer. Experimental results show that DualDecoder improves decoding throughput by up to 2.62$\times$ over state-of-the-art systems while preserving decoding latency and model quality.

DOI arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

OasisKV is presented, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding and observes that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD).

Can Xiao, Sukmin Cho, J. We et al. · 0 citations

#large language models Book Open access Aug 2026

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching

NeuroPrefetcher is presented, a storage-backed LLM inference system that exploits that MLP activity during autoregressive decoding has strong temporal locality, and achieves 7.9-12.0x speedup over llama.cpp across constrained memory budgets.

Nobel Dhar, Md Romyull Islam, Xuechen Zhang et al. · 0 citations

Book Open access Jul 2026

Scaling Attention Beyond GPUs for LLM Inference

Beyond is presented, a drop-in runtime that integrates a smart offloading scheme to selectively identify and retain salient KV entries across continuous decoding sessions, together with a hybrid CPU–GPU attention mechanism for scalable inference.

Weishu Deng, Yujie Yang, Peiran Du et al. · IEEE International Symposium... · 1 citation

Book Open access Jul 2026

PKAS: Predictive KVCache-Aware Scheduling for Faster LLM and Transformer Inferences

With rising popularity of LLMs, the performance, scalability, and resource-efficiency of inferences become a crucial challenge. The core part of the inference process is the KV cache, which avoids recomputing intermediate attention states, and the batching strategy that batches multiple requests per forward pass to leverage GPU parallelism. KV cache memory grows linearly with sequence length and batch sizes, easily exceeding the limited GPU memory capacity. State-of-the-art inference runtimes use continuous batching to maximize GPU utilization by interleaving the processing of new requests (i.e., prefill requests) with ongoing generation requests (i.e., decode requests). However, existing schedulers greedily admit prefill requests without considering the future KV cache memory required to successfully run the decode phases. This shortsighted approach causes frequent KV cache overflows, which in turn trigger preemption and recomputation of requests, severely degrading both throughput and latency. We propose PKAS, a Predictive KV Cache-Aware Scheduling algorithm to mitigate this inefficiency by reducing preemptions. PKAS uses a low-overhead technique to simulate future KV cache utilization and guide the admissibility for new request candidates. Combined with lightweight output-length predictions, PKAS can make better batching decisions, preventing KV cache overflows and drastically reducing preemptions. Evaluations on diverse models and workloads show that PKAS achieves up to 7.34x higher throughput and 8x lower latency compared to state-of-the-art scheduling, with the largest gains on long-context workloads where KV cache pressure is high.

Jie Ye, Avinash Maurya, Krishna Teja Chitty-Venkata et al. · IEEE International Symposium... · 1 citation

Jul 2026

Elastic Memory Remapping for Multi-tenant LLM Serving

This work presents Oneiros, a dynamic remapping engine for multi-tenant LLM serving that dynamically repurposes GPU memory allocated for model parameters as KV cache capacity, enabling nonblocking, unidirectional parameter transfer.

Ruihao Li, Shagnik Pal, Vineeth Narayan Pullu et al. · ACM SIGOPS Operating Systems... · 0 citations

Jul 2026

InferScale: GPU-Native KV Injection for Personalized LLM Serving

This work presents InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state, and encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV.

Peter Li, Prashant Pandey · arXiv.org · 1 citation