Retrieval Capacity of Self-Attention Under Competition
How many tokens from its context does a language model actually use, and what determines that number? We study this question through self-attention. Without retraining, we retain only the tokens with the highest attention weights at each head, layer, and query, keeping their original weights unchanged. By varying the s...
Timur Mudarisov, M. Burtsev, Radu State
· 0 citations