Lin-Feng Dong

KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression

As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing methods fall into three families: score-based eviction, summary compensation, and offload-and-recall. Yet all three decide what to keep or recall by content relevance to t...