Cross-request KV caching reduces the prefill cost of Retrieval-Augmented Generation (RAG), but conventional prefix caching severely limits cache reuse across requests. Position-Independent Caching (PIC) removes this constraint by reusing independent chunks, but their KV states miss cross-chunk interactions. Existing me...
Ruo-Ling Qi, Yi-Rui Liu, Xuan'er Wu et al.· 0 citations
SuffixReplay is presented, the first prefix caching system that lets hybrid LLMs reuse cached prefixes at every cache-supported page boundary without materializing recurrent-state checkpoints and is evaluated on three hybrid LLMs.
Yi-Rui Liu, Ruo-Ling Qi, Xuan'er Wu et al.· 0 citations
LLM serving is increasingly accelerated by position-independent caching (PIC). Existing PIC methods, however, are built for full-attention models, where a token-indexed KV cache underlies its core operations: matching reusable token chunks, concatenating their KV entries, and selectively recomputing a few tokens to res...
Yi-Rui Liu, Ruoling Qi, Long-Wen Wang et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.