AttnFusion: gated cross-attention for fine-grained visual injection in latent diffusion models

Abstract

In the domain of text-to-image generation, diffusion models have demonstrated remarkable creativity. However, they continue to face challenges such as "semantic misalignment" and "detail loss" when tasked with generating images that require specific, fine-grained visual characteristics. Current approaches frequently resort to basic feature stacking or tuning within a single modality; however, these techniques often fail to achieve a delicate balance between embedding intricate reference textures and maintaining the original text-driven semantics. To address this, we propose AttnFusion, a visual feature injection framework based on gated cross-attention. We present a streamlined Attentive Fusion Network that employs text embeddings as queries to actively harvest pertinent cues from OpenCLIP-encoded visual data. To prevent the visual signal from overshadowing the text, we implement a zero-initialized, learnable Context Gating module that autonomously regulates the blending of both modalities. Empirical evaluations on the challenging Food-101 benchmark demonstrate that AttnFusion markedly surpasses leading baselines, including DisenBooth, DreamBooth, and LoRA. It achieves an FID of 48.44, while maintaining superior text-image alignment (CLIP-T score of 29.15), demonstrating its robustness in both visual fidelity and semantic consistency.