Darío C. Larese

Flow-based Spectral Kernel Learning for Nonstationary Attention

Flow-based spectral kernel attention (FSKA), a framework for learning expressive nonstationary attention kernels, is proposed and it is shown that expressive kernel learning enables a simplified attention design with removed query--key projections and fixed orthogonal value projections, resulting in a more parameter-ef...