Shuangwu Chen

SpecCache: Speculative KV Cache Reuse for Efficient RAG Serving

This work finds that the deep-layer features of a lightweight speculative model exhibit strong consistency with the target model in the selection of critical tokens for recomputation, and proposes SpecCache, which employs deep-layer hidden-state norms from a speculative model as a proxy to guide the critical token selection for target large model.