Text-to-Image Generation Based on Multimodal Fusion

Abstract

Text-to-image (T2I) generation refers to the technology that takes natural language descriptions as input and automatically generates images semantically aligned with them, which is one of the core tasks in the field of cross-modal content generation. However, existing technologies still face core challenges in handling complex semantics, restoring fine details, and balancing generation diversity and controllability, making it difficult to fully meet the complex needs of diverse application scenarios relying solely on text prompts. To address these challenges, generation methods can be designed by improving multimodal fusion strategies based on different fusion approaches. For this purpose, from the perspective of multimodal fusion and with fusion timing as the main thread, this paper systematically sorts out mainstream T2I technologies into four main paradigms: early fusion, late fusion, attention mechanism-based fusion, and hybrid fusion. It systematically explains their core mechanisms, evolution trends, and conducts model performance comparisons. The paper further sorts out the key datasets supporting the development of the field, aiming to provide help and inspiration for subsequent related research in the field, identify potential research directions, and promote the practical application and innovative breakthroughs of text-to-image generation technology in more complex scenarios.