Attention Calibration for Position-Fair Dense Retrieval
A strength coefficient that interpolates between uncalibrated and fully equalized attention, together with an efficient implementation that reduces peak calibration memory overhead from 5-7 GiB to under 1 MiB, and a variant that preserves the pooling token's self-attention mass and redistributes only the remaining mass...
Andrianos Michail, Elias Schuhmacher, Juri Opitz et al.
· 0 citations