Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks
It is shown how pretrained information reshapes the implicit bias and training dynamics, and can thereby reduce the sample complexity of recovering the target parameters and support, by gradient-based fine-tuning.