Experiments show that HERO consistently improves long-horizon accuracy, stable rollout length, and out-of-distribution robustness at no inference-time cost, indicating that history-enriched relative supervision is effective for stabilizing long-horizon autoregressive prediction.
Abstract
Neural operators provide fast surrogates for time-dependent partial differential equations (PDEs) by applying a learned evolution operator recursively to its own predictions, but this autoregressive rollout feeds every prediction error back as input, so local errors accumulate. Existing rollout-training strategies reduce the mismatch between training inputs and self-generated states, yet their supervision still measures only the absolute discrepancy from the ground-truth trajectory. Such supervision is therefore uninformative about whether the operator has overcome the long-horizon failure behaviors it exhibited earlier during optimization. We propose history-enriched rollout training (HERO), which augments conventional absolute trajectory supervision with relative supervision derived from the model's optimization history. HERO ranks detached candidate rollouts from a periodically refreshed lagged operator, the current model, and a perturbed input by rollout error, spectral discrepancy, energy drift, and error growth, and selects the strongest failure trajectory as reference. This reference enters a margin-based objective as a fixed comparison baseline, inducing a bounded, sample-dependent reweighting of the ground-truth rollout gradient rather than an independent gradient direction, which we further analyze theoretically. Experiments on nine PDE benchmarks with spectral and attention-based backbones show that HERO consistently improves long-horizon accuracy, stable rollout length, and out-of-distribution robustness at no inference-time cost. These results indicate that history-enriched relative supervision is effective for stabilizing long-horizon autoregressive prediction.
This work forms AdamW as a finite-horizon input--state--output (ISO) system whose state contains the model parameters and first- and second-moment estimates, and derives an exact multistep error decomposition and establishes first-order finite-horizon accuracy under local smoothness and controlled activation switching.
Neural operators have shown strong potential for learning solution operators of partial differential equations (PDEs). However, long-horizon autoregressive prediction remains challenging: local errors accumulate as spectral inconsistency, phase misalignment, or mean drift. Existing methods mainly improve state representations and operator backbones, while leaving the repeatedly applied latent transition increment weakly structured, allowing spectral errors and unstable channel couplings to accumulate during rollout. To address these issues, we propose a geometry-aware incremental neural operator (GeoIncNO) for stable long-horizon PDE prediction. GeoIncNO predicts latent increments for residual advancement and uses lightweight low-rank projectors to regulate channel coupling within active frequency bands derived from the increment spectral energy distribution. To reduce physical-space reconstruction errors, GeoIncNO further introduces a mean--fluctuation decoupled reconstruction mechanism, where stable mean structures and dynamic fluctuations are fused separately, and phase correction is applied only to the zero-mean fluctuation component. Extensive experiments on six PDE benchmarks, covering 1D, 2D, and 3D dynamical systems, show that GeoIncNO achieves consistently strong prediction accuracy, improved rollout stability, and better spectral fidelity compared with competitive neural-operator baselines.
Jiaquan Zhang, Shuxu Chen, Haifan Meng et al.· 0 citations
Deploying machine learning surrogates in scientific simulations faces multifaceted challenges, primary among which is the lack of Continual Learning (CL) capabilities—specifically, the inability to adapt to new physical regimes without significantly degrading performance on prior ones. This is particularly problematic for autoregressive surrogates of time-dependent Partial Differential Equations (PDEs), where small prediction errors can accumulate over long rollouts and new physical regimes overwrite previously learned dynamics. We formulate this adaptation as a CL problem, demonstrating that while standard Experience Replay (ER) is a robust baseline across Advection-Diffusion, Burgers’, and Navier-Stokes equations, storing full high-resolution rollouts can be memory-inefficient. To address this, we introduce Replay-TS, a temporal-slicing replay strategy that stores compact autoregressive windows sampled across past simulations. Through empirical analysis, we show that Replay-TS exploits the low-frequency spectral redundancy of physical systems to enable sparse supervision for rollout steps. By preserving the contiguous historical context and sparsely penalizing the autoregressive target steps, Replay-TS improves retention performance under a fixed memory budget by leveraging higher sample diversity. Replay-TS consistently outperforms standard ER methods across standard 1D and 2D streams, achieving over a 30% MSE reduction in a mixed-physics stream, while remaining architecture-agnostic.
Hamed Hemati, Binh Duong Nguyen, Stefan Sandfeld· Machine Learning for Computa...· 0 citations
The Direct Prediction World Model is introduced, a non-recursive architecture that compresses an action sequence of arbitrary length into a single embedding and predicts the endpoint observation in a single forward pass, making long-horizon end-to-end training practical at horizons where unrolled autoregressive training becomes unstable.
Xinyi Li, Zaishuo Xia, Chenjie Hao et al.· 0 citations
Residual neural ordinary differential equations (NODEs) of the form f^=f+hθ can attain small one-step prediction error yet diverge under autonomous long-horizon rollout. A recent diagnostic attributes this to the one-sided Lipschitz (OSL) constant—the supremum over visited states of the logarithmic norm μ2(Jf^)=λmax(12(Jf^+Jf^⊤))—which, when positive, signals local expansion and amplifies persistent approximation error. In this work we convert this post hoc diagnostic into a train-time method by augmenting the one-step objective with a contraction penalty λEx[(μ2(Jf^(x))−c)+], and we study when this improves robust forecasting across stable, expansive, marginal, and chaotic regimes under realistic sensor-corruption noise. We prove that, at the penalty’s minimizer, the empirical OSL constant is controlled on the training set. A sample-to-domain covering condition then yields, via a Gronwall-type comparison, a conditional uniform-in-time rollout-error bound, with a time-averaged variant that justifies penalizing the mean rather than the maximum log-norm. Empirically, on a six-system, four-noise benchmark the penalty reliably drives the OSL constant down by one-to-two orders of magnitude, but whether this helps long-horizon accuracy is strongly regime-dependent and λ-sensitive: a common default (λ=0.1) over-damps and degrades rollout, whereas a calibrated λ≈0.01 helps only for measurably expansive baselines. Under a seed-decoupled re-evaluation with 15 seeds, the benefit is robust on a near-unstable, rotation-dominated oscillator—a 2.3× lower 100-step rollout error (p=0.018) at matched one-step error—but the apparent 5-seed improvement on a six-dimensional chemical reaction network does not replicate: with model and dataset seeds decoupled it reverses to a significant degradation, identifying the original effect as a seed-coupling artifact. The method thus yields a single robust positive result, and degrades already-contractive, conservative, and chaotic systems; for chaotic systems this is unavoidable, because enforced contraction suppresses the positive Lyapunov exponents that define the attractor. Finally, while a contraction-aware spectral penalty matches the log-norm penalty, standard ∥J∥2≤1 spectral normalization fails on rotation-dominated dynamics (8× worse rollout, p=0.0005, 15 seeds), confirming that the rotation-invariance of μ2 is the operative property.
Forecasting the long-horizon evolution of mechanical systems from position-only observations is a pivotal yet difficult task, as hidden velocities and trajectory-specific physical properties must be inferred simultaneously. Although physics-guided neural networks like Lagrangian Neural Networks (LNNs) guarantee physical plausibility, they generally require complete state inputs and lack adaptability to changing system parameters. To break these limitations, we introduce History-informed Lagrangian Neural Networks (HiLNN). Grounded in the insight that temporal position sequences implicitly encode underlying dynamics, HiLNN employs a recurrent encoder to extract a latent context from history. This context not only reconstructs the unobserved initial velocity but also adaptively modulates the mass matrix, potential energy, and damping coefficients of a structured Lagrangian system. By leveraging a differentiable RK4 rollout scheme, the entire pipeline is optimized end-to-end under multi-step trajectory supervision and energy-consistency regularization. Empirical evaluations across conservative, dissipative, and heterogeneous variable-parameter systems show that HiLNN delivers superior long-term prediction accuracy and maintains precise energy profiles compared to state-of-the-art baselines. The source code is publicly available at https://github.com/yingtian22/History-informed-LNN.
Tian-Shuo Zhang, Xianglei Xing, Wenzhe Zhai et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
