Jun 2026· arXiv.org· Vol abs/2607.20493· 0 citations· 24 references
Computer Science
TL;DR
This work introduces a novel framework for continual time series forecasting, designed to extend existing static forecasting models commonly used in the literature by incorporating an Experience Replay strategy guided by Attention mechanisms, which allows the model to adapt dynamically to new contexts while preserving prior knowledge, effectively mitigating catastrophic forgetting.
Abstract
Deep learning has led to remarkable progress in artificial intelligence, particularly in robotics, imaging and sound processing. However, a major limitation of neural networks remains their strong dependence on large and stationary datasets. In many real-world applications, these conditions are rarely met due to evolving and dynamic environments where data distributions change over time. Continual learning aims to address this challenge by developing models capable of adapting incrementally while maintaining a balance between stability and plasticity under computational constraints. In this work, we introduce a novel framework for continual time series forecasting, designed to extend existing static forecasting models commonly used in the literature by incorporating an Experience Replay strategy guided by Attention mechanisms. This approach allows the model to adapt dynamically to new contexts while preserving prior knowledge, effectively mitigating catastrophic forgetting. The framework is evaluated on standard forecasting benchmarks as well as on a piezometric dataset exhibiting diverse temporal behaviors. Results show that our approach effectively increases or maintains predictive performance over time while reducing retraining costs and data requirements, thus facilitating the deployment of forecasting models in dynamic and real-world settings.
An innovative approach to robustness in continual learning systems is introduced, the Dynamic Siamese Expansion Framework (DSEF) that employs a Siamese backbone architecture, comprising static and dynamic components, to facilitate the learning of both global and local representations over time.
Fei Ye, Yulong Zhao, Qihe Liu et al.· Neural Information Processin...· 3 citations
This review provides a structured overview of major CL paradigms, including task-incremental, domain-incremental, class-incremental, online, multimodal, and federated CL, and examines the theoretical foundations of CL, particularly the stability–plasticity dilemma, catastrophic forgetting, transfer dynamics, and representation learning.
Deploying machine learning surrogates in scientific simulations faces multifaceted challenges, primary among which is the lack of Continual Learning (CL) capabilities—specifically, the inability to adapt to new physical regimes without significantly degrading performance on prior ones. This is particularly problematic for autoregressive surrogates of time-dependent Partial Differential Equations (PDEs), where small prediction errors can accumulate over long rollouts and new physical regimes overwrite previously learned dynamics. We formulate this adaptation as a CL problem, demonstrating that while standard Experience Replay (ER) is a robust baseline across Advection-Diffusion, Burgers’, and Navier-Stokes equations, storing full high-resolution rollouts can be memory-inefficient. To address this, we introduce Replay-TS, a temporal-slicing replay strategy that stores compact autoregressive windows sampled across past simulations. Through empirical analysis, we show that Replay-TS exploits the low-frequency spectral redundancy of physical systems to enable sparse supervision for rollout steps. By preserving the contiguous historical context and sparsely penalizing the autoregressive target steps, Replay-TS improves retention performance under a fixed memory budget by leveraging higher sample diversity. Replay-TS consistently outperforms standard ER methods across standard 1D and 2D streams, achieving over a 30% MSE reduction in a mixed-physics stream, while remaining architecture-agnostic.
Hamed Hemati, Binh Duong Nguyen, Stefan Sandfeld· Machine Learning for Computa...· 0 citations
This framework is the first to address catastrophic forgetting by leveraging models in CL as their own memory buffers by exploiting an implicit bias of gradient-based neural networks due to which these converge to margin maximization points.
Pascal Janetzky, T. Schlagenhauf, S. Feuerriegel· Proceedings of the 32nd ACM...· 0 citations
Investigation of the influence of fully connected FC layer architecture on parameter regularization in the class incremental learning setting using a modified ResNet-18 trained on the CIFAR-10 dataset provides both a novel parameter regularization strategy and new insights into the interaction between network architecture and catastrophic forgetting in continual learning.
Henry Huang· Journal of high school scien...· 0 citations
This comprehensive survey formally defines the CTTA problem, analyzes the diverse continual domain shift patterns that characterize different evaluation protocols, and proposes a hierarchical taxonomy that categorizes existing methods into three families: optimization-based strategies (entropy minimization, pseudo-labeling, parameter restoration), parameter-efficient methods (normalization layer adaptation, adaptive parameter selection), and architecture-based approaches (teacher-student frameworks, adapters, visual prompting, masked modeling).
