MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
Across a wide range of latency and peak memory constraints, MetaKV consistently outperforms the best static configuration, improving constrained success rate (CSR), the fraction of prompts answered correctly while satisfying both constraints, by approximately 0.07 on average and up to 0.135.
Michael Wang, Keith Li, Roozbeh Bostandoost
· 0 citations