Skip to content

Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks

Sep 2026 · arXiv.org · Vol abs/2609.34756 · 0 citations · 37 references
Computer Science Mathematics

TL;DR

It is shown how pretrained information reshapes the implicit bias and training dynamics, and can thereby reduce the sample complexity of recovering the target parameters and support, by gradient-based fine-tuning.

Abstract

Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread practical success, how fine-tuning leverages information from pretraining remains poorly understood theoretically. We study fine-tuning from pretrained weights through the lens of sparse linear regression and two-layer diagonal linear networks. In our setting, pretraining provides information through the support (and signs) of the initialization predictor, which may contain coordinates relevant to the downstream task. We show how pretrained information reshapes the implicit bias and training dynamics, and can thereby reduce the sample complexity of recovering the target parameters and support. In particular, for a clean initialization with correctly inherited signs, we show that the required sample size is comparable to that of a weighted Lasso estimator that explicitly exploits the pretrained support through a suitably chosen regularizer. Our results thus show how information encoded in pretrained weights can be implicitly exploited by gradient-based fine-tuning, reducing the amount of data needed to recover a downstream task.

View source

Similar papers

#machine learning Preprint Sep 2026

Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent

This paper characterise the optimal split under data-dependent evaluation geometries induced by the fine-tuning problem as a compute-split problem under a two-stage pretrain--fine-tune procedure with fixed total optimisation budget, using regularised least squares trained by gradient descent as a tractable setting.

Alex Buna, Fang-Hui Liu, Patrick Rebeschini · 0 citations

Understanding LLM Parameter Update Sparsity through the Lens of Fisher

Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training...

Yu-Fan Zhang, Sagnik Mukherjee, Hao Peng · 1 citation
Aug 2026

SPIRA: Sparse Information-Geometric Rank Adaptation for Parameter-Efficient Fine-Tuning of Large Pretrained Models.

SPIRA is a PEFT framework that separates support discovery from parameter-efficient adaptation and remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.

Zhong-Yi Wen, Zhikai Zhai, Guo-Min Sun et al. · 0 citations
#machine learning Preprint Sep 2026

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

Extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.

Sankar Behera, Dhruv Singh, Anshika Agnihotri et al. · 1 citation
#machine learning Preprint Oct 2026

Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization

Large language models (LLMs) are typically pretrained in high precision but increasingly deployed with low-precision post-training quantization (PTQ). Recent studies have shown that using weight averaging during pretraining can improve PTQ performance compared with learning-rate decay, suggesting that it might provide...

Han Wang, Tianqi Shen, Zong-Lin Liu et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.