Abstract-LoRA: Unlocking Single-Image Style Transfer through Targeted U-Net Block Training

Abstract

Diffusion models represent one of the most advanced paradigms in generative modeling. Leveraging their development, a growing number of style transfer methods based on diffusion models have been proposed. However, among these methods, multi-image style transfer approaches that require at least five to ten style examples tend to achieve more satisfactory results. Single-image methods, by contrast, often struggle with either insufficient content preservation or inadequate style fidelity. This greatly limits style extraction from scarce artworks and undermines their artistic value. To address this, we propose Abstract-LoRA, a method that pushes the boundaries of single-image style transfer through lightweight LoRA training on specific U-Net blocks in diffusion models. Specifically, our work is inspired by B-LoRA, a style transfer method that achieves basic style-content disentanglement by training specific U-Net blocks. However, it suffers from a critical limitation: the inability to capture complex backgrounds. Building upon B-LoRA, our method conducts a more refined analysis of U-Net blocks, employing additional U-Net blocks and clustering-based abstraction of style images to better disentangle and balance style and content. Extensive experiments demonstrate that our proposed method not only generates visually more harmonious and satisfying artistic images but also quantitatively improves the preservation of both style and content in the final outputs.