Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers
In this paper, we propose Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a simple and effective attention mechanism that generalizes Multi-Head Attention (MHA) through the lens of metric learning. Standard MHA implicitly learns a static, head-wise Euclidean similarity corresponding to the diagonal blocks of a...
Zhen-Ke Duan, Xin Li, Ji-Qun Pan et al.
· 0 citations