Skip to content

BITS: Rethinking Fair and Comprehensive Evaluation for Irregular Time Series Forecasting

Sep 2026 · 0 citations · 43 references
Computer Science

TL;DR

BITS, a standardized, reproducible, and extensible benchmark for advancing research on irregular time series forecasting, is proposed and it is found that method performance varies substantially across irregularity characteristics, with no single modeling strategy consistently dominating.

Abstract

Despite recent progress in irregular time series forecasting, the field still lacks a unified benchmark for fair and comprehensive evaluation. Existing evaluations are often conducted on a limited set of datasets with inconsistent experimental protocols and predominantly error-based metrics, rendering it difficult to compare and assess methods fairly and comprehensively across diverse settings. To eliminate these limitations and accelerate progress, we propose BITS, a standardized, reproducible, and extensible benchmark for advancing research on irregular time series forecasting. BITS covers eleven datasets from nine domains with diverse irregularity characteristics, and it characterizes the datasets according to their missing rate, missing pattern complexity, sampling irregularity, and skewness. Further, it offers a unified pipeline for data preprocessing, model integration and evaluation, and reporting. It accommodates regular and irregular time series forecasting methods, including time series foundation models, under consistent settings, incorporating both error-based and non-error-based evaluation metrics. Findings include that method performance varies substantially across irregularity characteristics, with no single modeling strategy consistently dominating. We also find that using error-based or non-error-based metrics can yield different model rankings, highlighting the need for multi-dimensional evaluation. The code can be found at https://anonymous.4open.science/r/BITS-8F2E/.

View source

Similar papers

Conference Open access Oct 2026

A Multi-Dimensional Benchmark of Large Language Models for Time Series Forecasting

Large Language Models (LLMs) have recently been applied to time series forecasting via prompt-based inference, yet their evaluation remains limited by potential benchmark contamination and an overemphasis on accuracy. We introduce a systematic benchmark that evaluates LLMs across accuracy, efficiency, cost, and environ...

Zairo Bastos, Fernanda Scarcela, Wesley Barbosa et al. · 0 citations
Open access Nov 2026

MMCont-Net: toward full-scale context modeling for irregular series forecasting

Forecasting irregularly sampled multivariate time series (IMTS) remains a fundamental challenge in contemporary time series analysis, owing to the inherent non-uniformity of sampling and pronounced inter-variable asynchrony. While recent advances in context learning achieves remarkable performance, microscopic signal d...

Qing-Hua Ren, Yan-Lin Wu, Da-Long Gao et al. · 0 citations
Review Open access Dec 2026

A Comprehensive Survey of Time Series Forecasting: Concepts, Challenges, and Future Directions

Time series forecasting (TSF) is a fundamental task in diverse decision-making scenarios, such as business intelligence, energy management, healthcare, and scientific discovery. Over the past decades, TSF has been extensively studied, with methods evolving from traditional statistical models to recent data-driven deep...

Ming-Yue Cheng, Zhi-Ding Liu, Xiao-Yu Tao et al. · 21 citations
#machine learning Preprint Oct 2026

Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance

While established literature underscores the pivotal role of preprocessing in forecasting accuracy, this stage remains largely overlooked in current research. Modern benchmarks typically resort to simple scaling, failing to account for critical transformations required to address nonstationarity, such as differencing....

G. Padilha, P. D. de Mattos, Rafael M. O. Cruz · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.