Skip to content

SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting

Sep 2026 · arXiv.org · Vol abs/2609.20086 · 0 citations · 39 references
Computer Science

TL;DR

SETTer is introduced, a transformer-based model that addresses challenges of long-term multivariate time series forecasting by incorporating novel techniques for decoupled self-attention and hybrid masking and outperforms state-of-the-art models in 88% of the scenarios.

Abstract

Long-term multivariate time series plays a significant role in many application areas such as power systems, trading, etc. However, their accurate prediction is quite difficult for conventional forecasting methods as they often exhibit high dimensionality and complex relationships. Recent works show that transformer-based approaches are quite effective for long-term forecasting thanks to their attention mechanism. However, in the presence of complex high-dimensional inputs, they show evidence of oversmoothing, limited capacity, and opacity. To this end, this paper introduces SETTer, a transformer-based model that addresses these challenges by incorporating novel techniques for decoupled self-attention and hybrid masking. The proposed techniques enable SETTer to effectively capture the dominant short- and long-term patterns across the temporal and channel dimensions. In addition, we enrich the model layers with simple explainable structures that indicate the discriminative pattern of SETTer. We show that with a single-layer transformer architecture, SETTer can effectively model long-term dependencies in the presence of varying data complexities. Extensive experiments on real-word benchmark datasets for long-term multivariate time series forecasting demonstrate that SETTer outperforms state-of-the-art models in 88% of the scenarios.

View source

Similar papers

#graph neural networks Conference Sep 2026

A Deep Learning Framework for High-Dimensional Time Series Forecasting

A novel architecture that combines a graph neural network block with a Long Short-Term Memoryencoder-decoder block for end-to-end feature learning and forecasting in multivariate time series forecasting is proposed, resulting in accurate and reliable predictions.

Ali Sarabi, Arash Sarabi, G. Runger · 0 citations
Conference

Multi-Scale Transformer for Long-Term Time Series Forecasting

This paper proposes a novel architecture to improve forecasting performance over long horizons: the Multi-Scale Transformer, which effectively captures both short-term fluctuations and long-term trends by extending the transformer by integrating multi-scale processing.

YongKyung Oh, Alex A. T. Bui · 0 citations
Open access Sep 2026

Transformer Architectures for Multivariate Time Series Forecasting

This work starts from the underlying feature representation mechanism to deeply deconstruct data representation strategies which include dimensionality reduction decomposition, time-frequency domain conversion and masked pre-training, to provide a panoramic reference for the theoretical breakthroughs and engineering de...

De-Liang Zhang, Bo He, Yu Gao · 0 citations
Open access Sep 2026

FCA-Transformer: A Feature Pyramid Time Series Forecasting Model Driven by Cross-Attention Mechanism

The findings indicate that the proposed architecture successfully reconciles multi-scale feature extraction with lightweight dependency modeling, enhancing structural generalization and providing a scalable framework for real-time temporal analysis in complex industrial environments.

Lin-Li Wu, Ji-Yong Zhang, Zhi-Ming Zhang et al. · 0 citations
Conference Open access Sep 2026

StreamTimer: Efficient Inference for Long-Context Time Series Transformers

This work proposes an inference-only streaming autoregressive framework that replaces repeated full-context recomputation with a one-time context warmup and incremental decoding, enabling efficient long-history forecasting without retraining.

Xi-Yu Meng, Yuhan Wu, Can-Ran Xiao et al. · 0 citations
Open access Sep 2026

Deep learning forecasting model and visual explanation for large-scale time-series data

Large-scale time-series forecasting underpins decision-making in energy systems, traffic operations, industrial monitoring and financial analytics. Recent recurrent, convolutional and Transformer-based models have improved predictive accuracy, yet three limitations remain: inefficient modelling of long input windows, i...

Ya-Zhi Tao · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.