MoSAR: Mixture of Semantic Attention Regimes for Learning Adaptive and Approximable Attention Geometries
Mixture of Semantic Attention Regimes (MoSAR) is introduced, which learns such an adaptive, controlled-decay geometry over query--key interactions, suggesting that it is not only adaptive, but also amenable to low-cost approximation at inference time.
M. Paolicelli, Alessandro Petruzzelli, Alessandro Francesco Maria Martina et al.
· 0 citations