Perturbation Matters in Time Series Forecasting: A Wave-attention-aware Transformer Method

Abstract

Time series forecasting (TSF) refers to a fundamental task of predicting future sequential data based on historical observations. One representative category of TSF methods is transformer-based approaches, which translate time series into token sequences (i.e., as raw texts) before applying well-established transformers. In this paper, we conducted extensive preliminary experiments to evaluate their stability against various noises, and we empirically observed an interesting phenomenon: applying noises with appropriate levels to training time series can promote the predictive accuracy of transformer-based TSF methods. We analyze this phenomenon from the perspective of token attention and find one basic reason is that applying noises with appropriate levels can lead to token attention weights wavy, which is more consistent to the basic characteristic of time series data. Motivated by this phenomenon and analysis, we apply perturbations and propose a sub-objective wrt, perturbations constraining token attention weights to be wavy. Accordingly, we propose a novel TSF method, namely Wave-Attention-aware TransformER (WATER). We empirically evaluate \baby across the commonly used benchmark datasets, and experimental results indicate that it consistently outperforms the existing TSF methods.