More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding
Autoregressive generation in Large Language Models (LLMs) is constrained by the memory and computational demands of attention mechanisms. Sparse attention methods mitigate this cost by selecting only high-probability entries of the attention matrix. We observe that in many such methods, this renders the probability-val...
Noam Elata, Itay Lamprecht, Mikey Shechter et al.
· 0 citations