VQA-Guided Diffusion: Enhancing Text-to-Image Generation With Semantic Feedback From Visual Question Answering
TL;DR
It is shown that VQA can serve as an effective semantic feedback to significantly enhance prompt-image alignment without retraining the diffusion model, providing a powerful, interpretable and self-correcting strategy for text-to-image production.
Abstract
Diffusion models for text-to-image (T2I) synthesis, e.g. Stable Diffusion, generate visually realistic images, but often fail to capture the fine-grained semantic nuances of complex prompts. Typical failure modes include missing objects, incorrect attribute assignment e.g. wrong colors, distorted spatial relationships and inaccurate object counts. This paper proposes a VQA-Guided Text-to-Image Feedback Framework to address these limitations, substituting the traditional one-shot generation with an iterative self-correcting loop. In our framework, an input prompt is first decomposed into its semantic components (objects, properties, relationships) using natural language processing. Then a set of Visual Question Answering (VQA) questions is automatically generated stepwise to probe object existence, colors, spatial relations and quantities. A VQA model (BLIP) answers the questions on the initially generated image from Stable Diffusion. Whenever it detects a semantic mismatch, it automatically refines the prompt (for example, by adding explicit corrective instructions such as “ make sure the car is red ”), and feeds the revised prompt back into the diffusion model. This cycle is repeated until adequate semantic alignment is obtained, with the framework incorporating various assessment criteria (CLIP Score, VQA Accuracy, LPIPS, picture quality score, and consistency loss) integrated into a unified multi-metric evaluation framework. For our experiments on MS-COCO captions, DrawBench, and a custom set of challenging prompts, the proposed method resulted in the following improvements observed on our test sets: the average CLIP Score improved from 0.311 (baseline Stable Diffusion) to 0.364, a relative increase of 17%; VQA accuracy was above 87% for all prompts evaluated; and in a human preference study of 39 pairwise comparisons, the VQA-guided method was preferred in 76.9% of the cases. Ablation experiments demonstrated the beneficial contribution of many components of the framework. We show that VQA can serve as an effective semantic feedback to significantly enhance prompt-image alignment without retraining the diffusion model, providing a powerful, interpretable and self-correcting strategy for text-to-image production.