How many tokens from its context does a language model actually use, and what determines that number? We study this question through self-attention. Without retraining, we retain only the tokens with the highest attention weights at each head, layer, and query, keeping their original weights unchanged. By varying the s...
Timur Mudarisov, M. Burtsev, Radu State· 0 citations
A simple high-dimensional model is developed that separates the roles of norm, alignment, and endpoint geometry, showing how gradual directional changes can produce sharp reductions in competition and proving that a straight path toward the own endpoint cannot introduce new competitors under either Euclidean or cosine...
Timur Mudarisov, M. Burtsev, Radu State· arXiv.org· 0 citations
These results support a predictive-geometric view of transformer computation: near successful trajectories, the terminal loss induces a thin, anisotropic set of output-relevant hidden-state directions that can be measured and exploited for compression and distillation.
Timur Mudarisov, M. Burtsev, Tatiana Petrova et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.