On the Expressive Power of Transformers for Contextual Relations
This work develops a mathematical framework based on probability and optimal transport that provides a mathematical characterization of the expressive power of Transformers for contextual relations and shows how the choice of normalization determines the probabilistic structure of the relations represented by attention...