Application of a lightweight denoising diffusion model based on few-shot fine-tuning in handwritten digit generation

Abstract

Diffusion models excel at generative imaging but necessitate extensive datasets and considerable computational power. Few-shot learning with these models frequently results in overfitting and inconsistent generative quality. This study addresses these challenges for resource-constrained diffusion models under severe data scarcity by examining effective adaptation techniques. It propose a few-shot fine-tuning framework for unconditional generation, employing a bipartite training strategy predicated on the Tiny Diffusion architecture. Additionally, a dual-factor synergistic optimization mechanism, integrating a low learning rate (Low LR) with moderate data augmentation (MDA), is introduced and empirically validated to mitigate overfitting and catastrophic forgetting in few-shot scenarios. Evaluations were mainly performed on few-shot MNIST partitions (2-way 5-shot and 2-way 10-shot). Given MNIST's low-resolution grayscale images, a lightweight CNN feature space, trained on the dataset, was employed rather than conventional Inception-FID to compute FID-style distribution distance as a quantitative metric. This was complemented by ablation studies and comparative analyses across varying shot counts to precisely quantify the impact of sample size on the pre-trained model's generative output. Empirical evidence confirms non-linear performance improvements with increasing sample sizes. The proposed methodology significantly enhances the structural coherence and distributional fidelity of synthesized images. Specifically, the FID-style metric improved from 38.46±1.52 (2-way 5-shot) to 9.01±1.06 (2-way 10-shot), underscoring the substantial impact of increased sample size on pre-trained model generation capabilities. This research provides an efficient solution for deploying lightweight diffusion models in unsupervised few-shot generation tasks, demonstrating their viability in low-resource settings and offering practical implementation guidance and theoretical underpinnings.