Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs
SuffixReplay is presented, the first prefix caching system that lets hybrid LLMs reuse cached prefixes at every cache-supported page boundary without materializing recurrent-state checkpoints and is evaluated on three hybrid LLMs.
Yi-Rui Liu, Ruo-Ling Qi, Xuan'er Wu et al.
· 0 citations