Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking
Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN,...
Min Chen, Jun-Qin Wang, Ze-Guan Wu et al.
· 0 citations