Skip to content

Double Descent and Malign Overfitting in Diffusion Models

Sep 2026 · arXiv.org · Vol abs/2609.26392 · 0 citations · 68 references
Computer Science Physics

TL;DR

Overparameterization remains beneficial when paired with regularization: in the random-features theory and in U-Net experiments, optimally regularized large models---via a ridge penalty or early stopping, respectively---outperform any unregularized models.

Abstract

Conventional wisdom in deep learning holds that overparameterization---having more parameters $p$ than training samples $n$---is benign: larger models generalize better and, even without regularization, interpolating models generalize well, the test error following a double-descent curve. One might expect the same benign overfitting for diffusion models, whose training reduces to regression, i.e. to minimizing a quadratic score-matching loss. Yet the opposite is observed: overfitting here is catastrophic, driving the model into a memorization regime. We resolve this paradox by combining experiments on U-Nets trained on CelebA with a random-features model for which we derive closed-form learning curves. We show that with a fixed number $m$ of noise realizations per training sample, an interpolation peak does occur, but at $p\sim nm$ rather than at $p\sim n$ as in standard regression. The rise of the test loss, however, sets in much earlier, at $p\sim n$, independently of $m$. This overfitting is malign because, although the implicit regularization of training is fully at work, it drives the model toward the empirical score, which memorizes the training set, rather than toward the true score. A bias-variance decomposition pinpoints the mechanism: the bias of the score estimator starts to grow at $p\sim n$; past the peak the variance decays, as in regression, whereas the bias keeps growing and both saturate at a large value. Since diffusion models are trained with $m\gg1$, the peak is pushed to very large model sizes, and therefore sit on the rising branch that precedes it, where malign overfitting is already in play. Nevertheless, overparameterization remains beneficial when paired with regularization: in the random-features theory and in U-Net experiments, optimally regularized large models---via a ridge penalty or early stopping, respectively---outperform any unregularized models.

View source

Similar papers

Improved Distributional Diffusion Models

A stochastic few-step generator whose FID does not degrade as the sampling budget grows from 4 to 50 NFE, and the same recipe transfers to text-to-image generation.

Tommaso Martorella, Alexandre Galashov, F. Krause et al. · 1 citation
#machine learning Preprint Oct 2026

Memorization and Malign Generalization in Conditional Diffusion Models with Random Features

Conditional diffusion models generate diverse, novel, and high-quality samples under prescribed conditions. However, theoretical understanding of their memorization and generalization remains limited, while recent works have characterized these behaviors primarily in unconditional settings. In this work, we analyze a r...

Gwangho Kim, Sungyoon Lee · 0 citations

First Learn, Then Memorize: The Spectral Bias of Diffusion Models

Analytically, the spectrum is solved in the lazy high-dimensional limit for both linear ($n \asymp d) and polynomial ($n \asymp d^k$) sample complexities, and it is proved through a bias--variance decomposition that the first bulk minimizes the approximation error while the second drives the error associated with memor...

Raphaël Urfin, Tony Bonnaire, Giulio Biroli et al. · 1 citation
Preprint Oct 2026

Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks

Modern machine learning models are often overparameterized, i.e., have more parameters than training data. This typically implies that the loss function admits multiple minimizers so that training methods (and their hyperparameter choices) have significant influence on the computed solution. We study this so-called imp...

Wiebke Bartolomaeus, Holger Rauhut · 1 citation

Benign Overfitting for General Norms and Distributions

This paper develops a method to analyze benign overfitting in linear regression for general norms and general (sub-Gaussian) distributions and proves that minimum-p-norm interpolation with p>1 can benignly overfit even for non-Gaussian distributions, under suitable conditions.

Daniel Barzilai, Ohad Shamir · 0 citations
#machine learning Preprint Sep 2026

On the Two Faces of Adam in Separable Linear Classification

This work gives sufficient conditions under which an Adam-trained classifier nearly maximizes the max-norm margin before the updates become gradient-like, and shows that for polynomially decreasing stepsizes with exponent \(a), the updates become approximately proportional to the negative gradient after $\Theta(1/\epsi...

Fan Chen, Csaba Szepesvari · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.