Cross-modal alignment with language tokens for unified multilingual image-to-text generation

Abstract

Current multimodal translation systems typically employ a cascaded pipeline where image captioning and machine translation are performed sequentially, leading to error propagation and semantic drift. This paper introduces a unified neural architecture that integrates visual encoding, English caption generation and multilingual translation into a single end-to-end model. Our first key contribution is this unified framework which jointly optimizes both tasks through shared latent representations enabling richer cross-modal learning and more semantically consistent multilingual outputs than conventional systems. The architecture incorporates two core technical innovations. First, we introduce a cross-modal representation alignment mechanism that directly fuses hierarchical visual features from a CNN encoder with multilingual linguistic embeddings in the Transformer decoder. This ensures visual semantics (e.g., gender, count, and object attributes) are preserved during translation effectively grounding multilingual outputs in the visual context and rectifying a common failure mode of text-only NMT. Second, we design a language token driven conditioning strategy prepending learnable target language tokens to guide a single multilingual model. This allows dynamic adaptation to diverse syntax and morphology without requiring separate decoders for each language. Empirical evaluations on MS COCO and OPUS multilingual datasets demonstrate notable improvements over existing multimodal and cascaded baselines achieving a BLEU-4 score of 44.8 for English captioning and an average BLEU of 41.1 across five target languages (Chinese, French, Arabic, Spanish, and Hindi), representing gains of 2.2 and 2.0 points, respectively, over cascaded baselines. COMET scores averaging 0.732 further confirm the semantic fidelity of our visually grounded translations. The proposed framework establishes a scalable, visually grounded and linguistically adaptive solution for multilingual caption generation contributing a meaningful advancement toward integrated vision-language translation research and real-world accessibility applications.