CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
This work discovers that certain attention heads exhibit sequential consistency in their attention patterns, which can be persistently identified using a coefficient-of-variation-based algorithm, and proposes CateKV, a hybrid KV cache method that retains only critical token information for consistent heads, thereby reducing KV cache size and computational overhead.
Hao-Yun Jiang, Hao-Lin Li, Jian-Wei Zhang et al.
· International Conference on... · 2 citations