A Robust Image–Text Alignment Framework using BLIP-2, CLIP and Caption Guided Fusion

Abstract

Vision-language models have made remarkable improvements in multimodal understanding still, correctly mapping an image with its respective text is still problematic. A caption-based multimodal fusion scheme that fuses Bootstrapping Language-Image Pre-training Version 2(BLIP-2) and Contrastive Language–Image Pre-training(CLIP) to boost image-text alignment capabilities. This hypothesis lies on captions serving as semantic bridging for multimodal information. In the first stage, it utilizes BLIP-2 to generate a caption from the input image to extract semantic information at higher levels than pure visual information. Simultaneously, use CLIP to embed the image and text into a common space resulting in image vector and text vector. Subsequently, it encodes the generated caption through the CLIP text encoder to get semantic-aligned text vectors. Lastly, compute similarity scores between the image embeddings and text embeddings based on their semantic alignment. To increase representation power, this fusion block combines the semantic representation obtained from the caption generation process with the visual embeddings to allow a multimodal model to perceive both the visual and semantic information. This approach is evaluated on the Multi30K dataset, where image-text and text-image retrieval experiments were conducted. Conclusively, now it presents a scalable solution for improving multimodal alignment through the use of captions as a middle-level semantic representation.