TRALF: training-free regional prompt adaptation and layer fusion for transformer-based text-to-image diffusion models

Abstract

While recent Transformer-based diffusion models have significantly advanced text-to-image (T2I) synthesis, they inherently lack explicit spatial inductive bias. Consequently, generating complex scenes with multiple objects and fine-grained attributes often leads to severe "attribute leakage" and "spatial misalignment. " To overcome these limitations, we propose TRALF (Training-free Regional prompt Adaptation and Layer Fusion), a novel framework designed for Transformer-based architectures. TRALF achieves high-precision regional control without requiring any parameter fine-tuning. The framework consists of three synergistic modules: First, the Structured Semantic Planning Module (SSPM) utilizes Large Language Models (LLMs) to automatically parse complex prompts into structured spatial layouts. Second, the Spatially Decoupled Cross-Attention Module (SDCA) introduces a dynamic inference-time masking mechanism, enforcing physical isolation of cross-attention features to prevent semantic interference between non-target regions. Finally, the Global Consistency Fusion Module (GCFM) seamlessly integrates independent regional latent features with global background contexts using an adaptive weighted layer concatenation strategy, ensuring overall stylistic coherence. Extensive experiments on the T2I-CompBench benchmark demonstrate that TRALF significantly outperforms existing state-of-the-art models, including PixArt-α and SDXL, in attribute binding, spatial relationship adherence, and multi-object compositional generation. TRALF provides an efficient, plug-and-play paradigm for highly controllable T2I synthesis.