Knowledge-Aware Diffusion Models for Controllable Image Generation With Semantic Layout and Style Decoupling

Abstract

With the rapid development of diffusion models for digital image generation, achieving stable control over both semantic composition and visual style remains challenging. Existing approaches often rely on implicit attention or direct style injection, which may cause style–content leakage and unstable semantics. To address this issue, the authors propose Palette-Brush Diffusion (PBD), a controllable diffusion framework that decouples style modeling from semantic composition. The palette module models style distributions using whitening transformation and Gram statistics and injects them into diffusion features through affine modulation to preserve texture and color without copying structural content. The brush module separates structural and attribute information in text and maps structural embeddings into spatial layout weights that constrain cross-attention, thus establishing stable semantic–spatial correspondences during diffusion. Experiments demonstrate that PBD achieves more consistent style expression and more stable semantic composition than existing methods.