Positions Are Not Facts: The Mismatch Between KV Caches and Memory
When a fact changes, how should a language model update the history stored in its key-value (KV) cache? Hiding the old record is cheap, but it may still contain needed details or answer questions about the past. We compare hiding whole records, hiding only replaced values, and deleting old text and recomputing the cach...
Chang-Hai Zhou, Yu-Hua Zhou, Shi-Yang Zhang et al.
· 0 citations