SANTA++: Sampling Attention through Representative Keys
Attention often concentrates on a small subset of tokens in the context, but which subset matters changes from one query to the next. To exploit this changing structure, we introduce SANTA++, a training-free stochastic attention method that uses representative keys for memory-efficient selection without scanning the en...
Kyle Lee, Christian Z. Pratt, Ruo-Yu Fang et al.
· 0 citations