Revisiting the Role of Learned Attention Weighting in SASRec
TL;DR
A controlled SASRec variant is studied that replaces learned attention weights with uniform aggregation and is trained under an otherwise identical block structure and training recipe, and this modification often yields performance comparable to the original model, with clear dataset-dependent exceptions.
Abstract
Causal self-attention models such as SASRec are widely used in sequential recommendation, where learned attention weights are often assumed to provide crucial importance weighting over past interactions. Yet it is unclear when predictive performance truly depends on such non-uniform weighting. We study a controlled SASRec variant that replaces learned attention weights with uniform aggregation and is trained under an otherwise identical block structure and training recipe. Across fourteen benchmark datasets, this modification often yields performance comparable to the original model, with clear dataset-dependent exceptions. To explain this heterogeneity, we introduce a stage-wise norm-based decomposition that quantifies self-preserving vs. cross-position mixing within attention blocks. Across datasets, we find distinct regimes: low mixing yields robustness to uniformization; higher mixing tends to coincide with sensitivity, while some datasets exhibit substantial mixing without dependence on learned weighting. Our results provide a practical diagnostic for identifying when attention weighting is functionally utilized in sequential recommendation. The code is available at: https://github.com/keito0329/revisiting-sasrec.