Jul 2026· 2026 7th International Conference on Smart Systems and Inventive Technology (ICSSIT)· pp. 1816-1821· 0 citations· 19 references
Abstract
Image captioning models can produce rapid Sentences, without visual relationships, or insert non-existing plausible objects. A common cause is to compress image evidence into visual symbols that carry a weak neighbourhood context. The multimodal context-enhanced visual representation learning framework (MCVRL) addresses this error mode by adding local neighbourhood descriptors, global scene tokens, prefix-conditioned visual doors and adaptive contextual corrections be-fore caption decoding. The encoder is trained with cross-entropy and contrast terms for image–text alignment. MSCOCO 2014’s Karpathy test classification results show that BLEU-4, METEOR, CIDEr and SPICE are more powerful captioning bases. The best configuration received a score of 1.352 of the CIDEr compared to 1.308 of BLIP-2 in the same evaluation protocol. The results of the ablation show that the most important contribution is the visual feature enriched by the context, followed by crossmodal gating and adaptive contextual attention. Qualitative examples show that objects with hallucinations are fewer and that spatial relationships are better recovered.
Image captioning with pretrained vision and language models often requires substantial model adaptation, while lightweight settings restrict the number of components that can be updated. This study investigates an input-level visual conditioning approach for image captioning using a frozen ResNet-50 image encoder and a frozen Phi-3 Mini language model. A global image representation is first projected into ten continuous visual tokens. A gated local residual module is then introduced to incorporate spatial features from the final ResNet feature map without modifying either pretrained backbone. The framework was evaluated on the MS COCO Karpathy split under a fixed training and decoding protocol. The global visual-conditioning baseline achieved a CIDEr score of 0.8382 on the test split. Adding the gated local residual module increased CIDEr to 0.8693, while adding only 3.95 million stage-specific trainable parameters. Under a near-matched trainable-parameter budget, a rank-10 QKV-only LoRA control achieved a CIDEr score of 0.8439, whereas direct and gated local residual conditioning achieved 0.8611 and 0.8693, respectively. These results are limited to the evaluated frozen ResNet-50–Phi-3 configuration and do not imply a general advantage over LoRA variants. As an additional sequence-level optimization experiment, self-critical sequence training increased CIDEr marginally from 0.8693 to 0.8704 without changing the inference architecture. Caption quality was further examined using a structured LLM-based evaluation over all 5000 images in the Karpathy test split, assessing semantic accuracy, information coverage, fluency, and reference-based unsupported content. A zero-shot evaluation on Flickr30k showed a marked performance decrease, indicating limited cross-dataset generalization. Overall, the experiments provide an empirical analysis of global and local visual conditioning for parameter-efficient caption generation with frozen pretrained components.
Inspired by the linguistic phenomenon of code-switching, MMCS interleaves vision and language by replacing textual entities with their corresponding visual objects, enforcing local vision-language grounding.
Changhao Xiang, Shangyu Xing, Zhen Wu et al.· 0 citations
Multimodal Large Language Models (MLLMs) have achieved remarkable progress in Visual Question Answering (VQA), yet they continue to struggle with questions requiring precise spatial reasoning and fine-grained visual understanding. These limitations often manifest as object, attribute, and spatial hallucinations, where models generate confident but visually unsupported responses due to insufficient region-level and fine-grained visual grounding. To address this challenge, we propose ReVA, a region-aware VQA model that employs a frozen CLIP ViT-L/14 Vision Transformer (ViT) and a Qwen2.5-7B-Instruct large language model (LLM) connected through a dual bridge that aligns both whole-image and region-level representations with the LLM's embedding space. The image bridge maps final transformer block features into image tokens. The region bridge maps cropped features from enriched intermediate features across ViT blocks so early texture and later object cues are more evident, into K region tokens for every bounding box. ReVA uses a detector stack that supplies automatic zero-shot bounding boxes that are both question-agnostic and question-dependent, using RAM++ (Recognize Anything Model), spaCy, and Grounding DINO. The image tokens and region tokens are concatenated as an LLM prompt prefix to jointly encode scene-level context and fine-grained regional evidence when answering questions. Evaluated on VQAv2, MMBench, POPE, and SEED-Bench, ReVA achieves 82.85% mean F1 on POPE, compared with 81.14% for an image-token baseline without region tokens. These results demonstrate that explicit region-aware visual representations reduce object hallucination and improve the factual grounding of MLLMs.
This work extends its previous work in scene graph generation to integrate image captioning, incorporating large language models to interpret scene graph structures and produce high-quality captions encompassing complex actions and relationships.
Anfel Amirat, N. Baha, Lamine Benrais· Signal, Image and Video Proc...· 0 citations
Visual Question Answering (VQA) requires effective cross-modal reasoning between visual content and natural-language questions. This challenge is particularly significant for Vietnamese due to the relatively limited availability of annotated VQA resources. This study proposes a dual-stream architecture that exploits complementary Transformer-based and convolutional visual representations for Vietnamese VQA. A Vision Transformer (ViT) is employed to obtain globally contextualised visual features, while ConvNeXt V2 preserves spatially structured visual information. Questions are encoded using PhoBERT. Instead of directly merging the two visual streams, the proposed model uses the textual representation as a shared query in two independent cross-attention modules, allowing question-relevant information to be retrieved separately from each visual representation prior to fusion. The resulting representations are concatenated along the sequence dimension, pooled, and combined with the sentence-level textual embedding through a residual connection for answer classification. Experiments on the ViVQA benchmark show that the proposed architecture achieves 64.31% accuracy and 62.65% F1-score. Additional experiments across multiple ConvNeXt V2 backbone scales consistently show improvements over the corresponding single-stream convolutional baselines, supporting the complementary role of the two visual representations.
Huy Tran, V. Nguyen· Asian Journal of Research in...· 0 citations
A unified multimodal search framework that integrates generative AI-based captioning and image understanding for improved retrieval, enabling more accurate, context-aware search in applications such as e-commerce, multimedia, and large-scale retrieval.
Saeed Alzahrani, Farah Mohammad, Nazar Hussain· Journal of Organizational an...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
