Do Artificial Neural Networks Need Dendritic Nanotubes? Two Reasons for Shrinking Importance Estimates in Continual Learning

Abstract

Regularization-based continual learning protects each parameter in proportion to an estimate of its importance, yet these estimates come from finite samples and heuristics and are used as if they were exact. We report four studies that start from a biologically inspired mechanism and end with a statistical explanation of it. Motivated by dendritic nanotubes, we let transient lateral links diffuse importance tags between hidden units. Links between co-active units gave no significant gain, but random links with lifetimes of 25 or 200 steps raised final accuracy on 20-task Permuted MNIST by 1.5 to 2.6 points. We show that random diffusion implements linear shrinkage of importance toward the layer mean in expectation. A simulation of the link process alone predicts shrinkage weights of 0.23 and 0.52 for these lifetimes, and graph-free shrinkage at 0.25 and 0.5 reproduces both the accuracy and the dispersion of the consolidated importance, while every activity-dependent rule did worse than random links. A five-seed screen across EWC, SI, and MAS found nominal gains for SI and MAS on Permuted MNIST and losses on task-incremental Split CIFAR-100, where forgetting was small and shrinkage cost plasticity. Varying the number of importance samples then separates two mechanisms on Permuted MNIST. For EWC, the gain fell by about ten points per tenfold increase in samples, from 19.6 points with 10 samples to -0.4 with 1000, a variance effect. For MAS, a gain of about one point persisted at 1000 samples, consistent with a bias in the importance measure rather than noise. Shrinkage had no detectable effect for MAS on a domain-incremental CIFAR-100 benchmark. Finally, we show that a standard heteroscedastic empirical Bayes rule shrinks the largest Fisher-type estimates most, which may explain its poor results, and point to a log-scale alternative.