KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

Shuoshuo Wang Fang Xi Wen-Yuan Huang Qing Wang Junming Su

Jul 2026 · arXiv.org · Vol abs/2607.24260 · 0 citations · 41 references

Computer Science

TL;DR

This work proposes Knowledge Access Planning (KAP), a paradigm-shifting execution abstraction that elevates structured knowledge priors from passive prompt-construction hints into first-class physical execution artifacts, and derives a phase-boundary model for the positive-speedup regime of plan-guided execution.

Abstract

Modern LLM systems increasingly rely on knowledge-selection processes that produce high-value structured priors, such as ranked evidence, graph topology, multimodal alignment, and confidence signals. Yet LLM serving remains fundamentally oblivious to this rich structure: once such signals are serialized into a prompt, the backend observes only a flat token sequence, forcing dense and uniform consumption of the full key-value (KV) state during decoding. We term this architectural mismatch the Knowledge Selection-Runtime Consumption (KSRC) gap: richer contexts enlarge the full-prompt KV footprint and decode-time memory traffic, increasing latency and degrading throughput even when reasoning depends on only a small fraction of the context. To bridge the gap, we propose Knowledge Access Planning (KAP), a paradigm-shifting execution abstraction that elevates structured knowledge priors from passive prompt-construction hints into first-class physical execution artifacts. KAP establishes a universal intermediate representation (IR)-the runtime access plan-which compiles structured knowledge signals to govern physical KV access without altering logical prompt semantics, model weights, or training procedures. Through this IR, KAP shifts LLM serving from token-aware context consumption to plan-driven, knowledge-aware runtime consumption. We instantiate KAP with GraphSpec, a compiler-executor realization connecting structured knowledge selection to an LLM serving backend. We derive a phase-boundary model for the positive-speedup regime of plan-guided execution. Across 4K-128K long-context QA workloads, GraphSpec maintains answer quality comparable to full-context decoding while decoupling physical KV consumption from prompt length, reducing proposal-time KV access to 5.5% of source KV state at 128K, and fundamentally shifting the scaling trajectory of long-context generation.

DOI arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

This paper introduces a mean correction term that effectively suppresses the approximation error, keeping performance degradation manageable even at extreme sparsity levels, and redesigns the sparse attention operator with PackGQA memory access, warp specialization, and pingpong pipelining.

Qihang Fan, Huaibo Huang, Zhiying Wu et al. · 0 citations

Preprint Jul 2026

SemPIC: Learning Semantic Position-Independent KV Caches

Long-context retrieval and agentic workloads repeatedly reuse the same documents under changing instructions, histories, and document orders. Prefix caching cannot exploit this reuse, while position-independent caching (PIC) remains unreliable because independently compiled KV states lack the future context in which they will be consumed. Our diagnostics show that a learned boundary-conditioned baseline sharply reduces attention deviation near reusable-block boundaries but leaves interior and task-level residuals, motivating adaptation of the document representation itself. We present \emph{SemPIC}, which trains a LoRA-enabled Writer to compile native per-layer document KVs through behavioral distillation while retaining the pretrained decoder as an unchanged Reader. Adaptation is confined to offline cache construction, preserving the standard KV interface and cache-hit decoding path. We further introduce KV Gradient Checkpointing, which reduces peak training memory without severing gradients through cached KVs. Across three models and four tasks, SemPIC raises mean micro-F1 over KV Packet from 0.53 to 0.60, approaching Full Recompute at 0.62. Code: https://github.com/jn12-29/SemPIC

Hui Xie, Peng Xiao, Yutong Deng et al. · 0 citations

Preprint Aug 2026

RAC: Reference-Aware Activation Compression for Communication-Efficient Split LLM Inference

System is presented, a reference-aware codec that retrieves exact-token historical spans for prefill uplinks, reuses the reconstructed uplink state for same-round prefill downlinks, and generates boundary-specific decode references with lightweight causal predictors.

Guotao Yang, M. Zhao, Haopeng Li et al. · 0 citations

Conference Jul 2026

Pegasus: Accelerating Large Language Model Inference with Stateful Prefix Caching

Modern large language model (LLM) inference suffers from severe Time-To-First-Token (TTFT) bottlenecks. Existing prefix KV caching mechanisms are inherently stateless, forcing a trade-off between cross-chunk attention accuracy and online recomputation overhead. To address this issue, we propose Pegasus, a novel stateful prefix KV caching system that aims to achieve full-context attention accuracy while avoiding costly recomputation. To handle the exponential growth of context states under limited memory capacity, Pegasus employs a Recursive Path-Pruning Caching (RPPC) algorithm to selectively cache high-value states based on access frequency, memory footprint, and asymmetric latency benefit. In addition, Pegasus introduces a transition-based KV management mechanism to mitigate cache-miss overhead. By exploiting the sparsity of state-dependent KV variations, it replaces expensive attention recomputation and I/O-intensive tensor reloading with lightweight sparse state transitions. Extensive experiments show that Pegasus improves end-to-end serving throughput by 45.9% on average, reduces TTFT by up to 78.5%, and lowers cache-miss recovery overhead by more than 72%.

Fahao Chen, Peng Li, Dongxiao Yu et al. · Fall Joint Computer Conferen... · 0 citations

Preprint Aug 2026

DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference

This work proposes DistillCache, a reinforcement learning framework that formulates KV-cache eviction as a sequential decision problem and demonstrates the effectiveness of learned, distribution-aware policies for memory-efficient long-context LLM inference.

Asaad Althoubi · 0 citations

Expanding the Physical Design Space of LLM Data Systems

It is argued that LLM-native data systems should expose selected inference-time mechanisms to the optimizer as physical design choices and advocate constructing Pareto frontiers of candidate implementations and exposing only non-dominated choices to the optimizer.

Gabriele Sanmartino, Matthias Urban, Carsten Binnig et al. · 0 citations