Projective Shannon Sketches for Transformer KV Memory: Needle Rescue under Compressed Keys on GPT Model
Projective KV-Cache: Slashing LLM Memory Costs by 50%–75% Without Context Amnesia (Empirical Validation on GPT-2) The Executive Problem: Large Language Models (LLMs) processing long documents face an expensive bottleneck: the Key-Value (KV) Cache. Storing every token's attention memory in GPU VRAM creates immense hardw...