Skip to content

A Lightweight Plug-in Gate for Transformer-Based Time-Series Forecasters

Sep 2026 · 0 citations · 21 references
Computer Science

TL;DR

The interface is implemented with a lightweight representation-level pre-encoder gate that assigns sigmoid scores to representation units, and the usage penalty reduces average admission scores while keeping forecasting errors close to the unpenalized TimeXer setting.

Abstract

Covariate-rich time-series forecasting requires deciding how external variables enter the target forecasting path. Existing Transformer-based forecasters usually build a covariate representation and pass it to the encoder without an explicit admission stage. This paper studies pre-encoder covariate admission as an input-side interface that regulates that representation immediately before encoder processing. We implement the interface with a lightweight representation-level pre-encoder gate that assigns sigmoid scores to representation units, and we also study a usage-regularized variant that penalizes average admission. The interface is evaluated as a plug-in module for TimeXer, Inverted Transformer (iTransformer), and Patch Time Series Transformer (PatchTST) under a zero-extra-tuning protocol, where each gated model inherits the corresponding baseline configuration. Experiments on the Electricity Transformer Temperature minute-level (ETTm1 and ETTm2) datasets, Traffic, Energy, and influenza-like illness (ILI) include paired forecasting comparisons, gate-placement ablation, initialization ablation, controlled covariate-admission analysis, and a variance inflation factor (VIF)-informed permutation feature importance (PFI) diagnostic case study. In the tested settings, the gate is competitive with the corresponding baselines, and the usage penalty reduces average admission scores while keeping forecasting errors close to the unpenalized TimeXer setting.

View source

Similar papers

#machine learning Preprint Oct 2026

CARE: A Lightweight Plug-in Gated Correction and Uncertainty-aware Module for Long-term Time Series Forecasting

Multivariate long-horizon forecasting is critical to electricity load scheduling and traffic flow management, and to financial risk control. Existing deterministic backbones output a single trajectory, masking heterogeneous prediction difficulty across horizons and channels and providing no localized reliability signal...

Guo-Pu Cheng, Chang-Long Lv, Jing-Yi Hou · 0 citations
Open access Nov 2026

Bi-Mamba-Load: A bidirectional structured state space model with sequence relationship adaptive perception for electricity load forecasting

The accurate prediction of electricity load is a critical component for smart-grid management, energy allocation, and grid stability. Traditional statistical methods and Convolutional Neural Networks (CNNs) often struggle with nonlinear relationships or long-range temporal dependencies, whereas Transformer architecture...

Jun-Wei Zhu, Ling-Zhi Xi, Chong-Yang Liao et al. · 0 citations
Conference

Multi-Scale Transformer for Long-Term Time Series Forecasting

This paper proposes a novel architecture to improve forecasting performance over long horizons: the Multi-Scale Transformer, which effectively captures both short-term fluctuations and long-term trends by extending the transformer by integrating multi-scale processing.

YongKyung Oh, Alex A. T. Bui · 0 citations
Conference Nov 2026

PatchTST-based short-term load forecasting with sliding-window residual correction for distribution networks

Short-term load forecasting plays a vital role in day-ahead scheduling, demand response, and energy-storage dispatch in distribution networks. The common recurrent neural networks always present lagged response around the peak-valley transition points and abrupt load change points, and the traditional transformer alway...

Yue-Lin Zou · 0 citations
Open access 2026

Evaluating Temporal Fusion Transformers for Short-Term Load Forecasting With Peak-Aware Adaptive Bias Correction

Short-Term Load Forecasting (STLF) is essential for maintaining a stable, affordable, and sustainable electricity supply across residential, commercial, industrial, transportation, and public sectors. Deep learning and ensemble methods such as Long Short-Term Memory (LSTM) and Extreme Gradient Boosting (XGBoost) have e...

A. Sanfilippo, A. Boumaiza, F. Ahmad · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.