Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets
This work reformulates eviction as a deployment risk-control problem: a material degradation occurs when eviction lowers task utility by more than a deployment-specified tolerance relative to full-KV inference on the same request, and deployment risk is the population frequency of such events.
Beomgu Kang, Sojin Yun, Hojoon Kim et al.
· 0 citations