Navigating parameter space: mitigating catastrophic forgetting in continual learning
TL;DR
Investigation of the influence of fully connected FC layer architecture on parameter regularization in the class incremental learning setting using a modified ResNet-18 trained on the CIFAR-10 dataset provides both a novel parameter regularization strategy and new insights into the interaction between network architecture and catastrophic forgetting in continual learning.
Abstract
Artificial general intelligence requires neural networks to continually acquire new knowledge while preserving previously learned information. However, sequential learning causes catastrophic forgetting, whereby performance on earlier tasks deteriorates as new tasks are learned. Parameter regularization methods, such as Elastic Weight Consolidation (EWC), attempt to address this challenge by constraining updates to parameters deemed important for previous tasks. This work investigates the influence of fully connected (FC) layer architecture on parameter regularization in the class incremental learning setting using a modified ResNet-18 trained on the CIFAR-10 dataset. Experiments demonstrate that a dynamically expanding FC architecture substantially outperforms a conventional static FC layer. To address the limited memory retention observed with EWC in static FC architectures, a new parameter regularization approach, termed Range Matrix, is proposed. Unlike EWC, which selectively constrains parameters identified by the Fisher Information Matrix, Range Matrix constrains all network parameters within dynamically computed bounds derived from loss gradients. Experimental results show that Range Matrix improves memory retention and reduces catastrophic forgetting relative to EWC when using a static FC architecture. Control experiments further demonstrate that the improved performance of Dynamic FC arises primarily from context partitioning rather than increased model capacity alone. Finally, analysis of the overlap between Fisher-important parameters and Range Matrix-constrained parameters shows that the two methods identify substantially different parameter subsets despite achieving similar performance under Dynamic FC, suggesting that FC architecture is the dominant factor governing continual learning performance. These findings provide both a novel parameter regularization strategy and new insights into the interaction between network architecture and catastrophic forgetting in continual learning.