Skip to content
Preprint

Learning without Overwriting: Structural Knowledge Fusion for Continual Learning

Oct 2026 · 0 citations · 35 references
Computer Science

TL;DR

The Knowledge Fusion Network (KFN), which mitigates the stability-plasticity trade-off in class-incremental learning through novelty-gated structural expansion rather than parameter negotiation within a shared capacity budget, establishes novelty-gated expansion as a practical path beyond fixed-capacity architectures.

Abstract

Catastrophic forgetting in sequential deep learning arises from a fixed-capacity architectural constraint: parameter updates encoding new tasks perturb weight subspaces supporting prior representations. We introduce the Knowledge Fusion Network (KFN), which mitigates the stability-plasticity trade-off in class-incremental learning through novelty-gated structural expansion rather than parameter negotiation within a shared capacity budget. KFN decouples task acquisition from knowledge consolidation: each incoming task is learned in an isolated lightweight specialist network (2.8M parameters) and integrated into the global model via a Structural Fusion Module that allocates dedicated channel capacity conditioned on a quantitative novelty score. Dual structural isolation--channel-space separation with temporally phased gradient unfreezing--protects consolidated knowledge. On ten-task Split CIFAR-100, KFN achieves 11.94% minimum task accuracy and 71.16% task-10 accuracy, whereas shared-parameter baselines degrade severely: LwF to 9.07% average accuracy (0.00% min), Replay++ to 8.91% (85.08% forgetting), and Replay to 16.21% (73.44% forgetting); KFN's forgetting (57.92%) is 15.5 pp lower than Replay. Furthermore, KFN prevents logit-scale drift, maintaining near-symmetric task performance. Across CIFAR-100 and EuroSAT satellite imagery, KFN's class-incremental retention (72.7% and 42.5%) substantially exceeds the best baselines (29.6% and 0.94%). Crucially, KFN achieves parameter-efficient linear scaling, adding 25% of the base parameter count per step and requiring 25.1% of naive backbone duplication over ten tasks (28.1M vs 112.0M). With a 37.1% peak GPU memory reduction over full retraining, KFN establishes novelty-gated expansion as a practical path beyond fixed-capacity architectures.

View source

Similar papers

#machine learning Preprint Oct 2026

Gated Target Propagation for Compositional Generalization in Continual Learning

Continual learning is typically framed as acquiring new knowledge without catastrophically forgetting previous tasks. However, a flexible continual learner should also be able to reuse and recombine previously acquired knowledge to rapidly solve novel task compositions. We introduce Gated Target Propagation (GaTaP), a...

Abdel Mfougouon Njupoun, C. Bredenberg, B. Richards et al. · 0 citations

Distribution-Conditioned Task Routing for Class-Incremental Learning

This work identifies three sources of routing error (feature-level, task-level, and class-level misalignment) and proposes Feature Distribution Calibration (FDC), which improves final accuracy in all 40 method-dataset pairs by 4.39 percentage points on average.

Long-Huan Xu, Zhi-Peng Zhou, Wei-Liang Ji et al. · 0 citations
#machine learning Preprint Sep 2026

CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework

This paper presents a novel sparsity-driven continual learning framework that operates in two stages: it first identifies a sparse, task-critical parameter mask via a sparsity-inducing objective, then performs mask-constrained fine-tuning by only optimizing parameters selected by the mask.

Yun-Xiang Fu, Meng Lou, Zicheng Liao et al. · 1 citation
#artificial intelligence Preprint Oct 2026

HERD: Harness-Evolving Recursive Distillation for Continual Learning

Continual learning requires a model to keep acquiring new skills and knowledge without compromising its existing capabilities. Parametric methods suffer from catastrophic forgetting and loss of plasticity. Non-parametric methods, such as prompts, skills, and harnesses, leverage in-context learning but are limited by co...

Xu-Rui Sun, Thanh Thi Nguyen, Thuy-Trang Vu · 1 citation
#machine learning Preprint Oct 2026

SPARS: A Stability-Plasticity-Aware Region Selection Framework for Continual Learning

Continual learning aims to acquire new knowledge while retaining previously learned knowledge. However, updating shared parameters when learning new tasks often overwrites existing knowledge, causing catastrophic forgetting. Memory replay methods mitigate catastrophic forgetting but rely on previous samples, raising pr...

Teif Aldaajani, Huda Hakami, Maha A. Thafar · 0 citations

Width Expansion as a Method for Class Incremental Learning

This work proposes a dynamic width expansion method that increases the number of neurons within existing layers according to a normalized loss criterion, without requiring task-specific information.

A. 1. Conde, Y. Elkhatib, C. Ranieri · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.