Attention Integration Strategies in MLP-Based Stock Movement Prediction: Effects on Performance, Stability, and Interpretability

Abstract

Stock movement prediction remains challenging because financial data are non-stationary and noisy. While attention mechanisms are widely used to enhance neural networks, how different attention integration strategies affect performance and training stability has not been systematically examined. We present a multi-seed empirical analysis of MLP-based models under three integration strategies—plain MLP, naively inserted self-attention, and residual attention—together with LSTM and Transformer baselines, evaluating 16 configurations on 10 CSI 300 and 10 S&P 500 stocks under a strictly forward-looking label, a purged chronological split, and early stopping: 1600 runs in total. Interleaved naive attention, in which self-attention output replaces the forward-path representation, collapses to degenerate single-class predictions in every run on both markets, whereas strategies that preserve the original representation collapse no more often than attention-free models; this difference is decisive after correction for multiple comparisons (Holm-adjusted p < 10−3). In contrast, no significant accuracy differences are detected among families, including the baselines, once collapsed runs are excluded, and unconditional balanced accuracy is within 0.01 across all non-degenerate families—consistent with the limited short-horizon predictability implied by market efficiency theory. A Jacobian analysis of the attention block, verified by training diagnostics, attributes the collapse to a rank-one attention matrix that compresses the forward representation to a single scalar unless the original representation is retained. A multi-seed analysis further shows that attention weight interpretations are not reproducible across random seeds and should be validated across repeated runs. These findings show that integration strategy and representation preservation—not the mere inclusion of attention—determine whether attention-augmented lightweight models train reliably.