ConsistDiff: Training-free reinforcement learning for semantically consistent text-to-image generation.

Abstract

Diffusion models have become a leading paradigm for text-to-image generation owing to their promising capability to produce photo-realistic and diverse images from text prompts. Despite these advances, state-of-the-art models often fail to maintain faithful semantic consistency, particularly when handling complex descriptions with multiple entities and attributes. These shortcomings manifest as object omission, attribute leakage, or inaccurate binding between textual concepts and visual elements. To address these challenges, particularly for deploying generative models on resource-constrained devices, we propose ConsistDiff, an inference-time optimization framework that enhances Stable Diffusion without requiring computationally expensive fine-tuning. By formulating the denoising process as an online reinforcement learning task, we dynamically adjust token weights to improve semantic alignment. Specifically, we integrate a context-aware attention mechanism to refine inter-token relations by leveraging character-level spatial distances. This context-aware design provides a fine-grained structural signal that sharpens attribute-entity binding, suppresses spurious long-range couplings, and adds negligible computational overhead. Further, we formulate prompt-specific token reweighting as a reinforcement-learning problem and optimize it with an attention-separation reward that explicitly penalizes cross-entity attention overlap while preserving the saliency of key objects. Extensive experiments on multiple benchmark datasets demonstrate that ConsistDiff consistently improves semantic alignment between the input text prompts and the generated images while maintaining highly efficient inference as well as mitigating common issues such as attribute leakage and entity inconsistency.