Aug 2026· The Visual Computer· Vol 42· 0 citations· 52 references
TL;DR
A Low-Complexity Cross-Modal Alignment via Projection (LCAP) network is proposed, which introduces Projective Token Compression (PTC), which leverages Mish activation and adaptive average pooling to reduce feature redundancy while enhancing discriminative information, and Positional Spatial Enhancement (PSE), which explicitly injects positional cues into the compressed representations and strengthens spatial structure.
Multimodal deep learning integrates heterogeneous data sources such as images and text to enable machines to understand complex real-world contexts. Although recent vision-language models have achieved significant progress, most existing approaches rely on rigid fusion strategies that combine modalities either at early or late stages without dynamically adjusting modality contributions. Such strategies may become unreliable when one modality is noisy, incomplete, or less informative.This paper proposes an adaptive attention-based fusion framework that dynamically balances visual and textual representations. The proposed architecture incorporates bidirectional cross-modal attention together with an instance-level gating mechanism that determines the relative importance of each modality for every input sample. Visual and textual features are first extracted using modality-specific encoders and then aligned through cross-modal attention layers. Subsequently, a lightweight gating network assigns modality weights to construct an interpretable fused representationThe training objective integrates task-specific supervision, contrastive alignment, and regularization to prevent modality dominance. Experiments on VQA v2 and MS-COCO benchmarks demonstrate consistent improvements over static fusion approaches and representative vision-language transformer baselines. The proposed framework also exhibits robustness when one modality is missing and provides interpretable modality contributions.
Unnati A. Patel, Sanskruti Patel, J. Nanavati et al.· International Conference on...· 0 citations
Producing a reliable multimodal summary requires a model to determine which claims are supported across language and vision and then express that evidence coherently. Training only for generation leaves useful cross-modal supervision untapped, yet indiscriminate parameter sharing across auxiliary tasks can merge transferable semantics with task-dependent cues and thereby induce negative transfer. Alignment is made still harder by visually salient background regions that have little bearing on the desired summary. To address these difficulties, we develop a single framework for multimodal summarization, image-text matching, and image-text retrieval. Visual tokens are first weighted through text-conditioned semantic attention; a learned gate then suppresses unhelpful visual responses before fusion. The aligned sequence is decomposed into one common representation and three private, task-oriented representations. Their redundancy is controlled by a Frobenius-norm orthogonality penalty, optimized jointly with the generation, matching, and retrieval objectives. The implementation couples a T5-base language encoder with CLIP ViT-B/16 and maintains 768-dimensional features in both streams. On MSMO, the full system records ROUGE-1, ROUGE-2, ROUGE-L, and BLEU values of 46.95, 22.03, 44.42, and 18.71, respectively, exceeding the single-task and multitask comparators included in the reported evaluation. Removing adaptive alignment, the shared-private decomposition, or the orthogonal term lowers ROUGE-L by 2.91, 2.08, and 0.87 points. Taken together, the findings indicate that text-directed visual filtering and an explicit division between reusable and task-specific information are both consequential for grounded summary generation.
Zhiang Chen· AI and Data Science Journal· 0 citations
Visual Question Answering (VQA) requires effective cross-modal reasoning between visual content and natural-language questions. This challenge is particularly significant for Vietnamese due to the relatively limited availability of annotated VQA resources. This study proposes a dual-stream architecture that exploits complementary Transformer-based and convolutional visual representations for Vietnamese VQA. A Vision Transformer (ViT) is employed to obtain globally contextualised visual features, while ConvNeXt V2 preserves spatially structured visual information. Questions are encoded using PhoBERT. Instead of directly merging the two visual streams, the proposed model uses the textual representation as a shared query in two independent cross-attention modules, allowing question-relevant information to be retrieved separately from each visual representation prior to fusion. The resulting representations are concatenated along the sequence dimension, pooled, and combined with the sentence-level textual embedding through a residual connection for answer classification. Experiments on the ViVQA benchmark show that the proposed architecture achieves 64.31% accuracy and 62.65% F1-score. Additional experiments across multiple ConvNeXt V2 backbone scales consistently show improvements over the corresponding single-stream convolutional baselines, supporting the complementary role of the two visual representations.
Huy Tran, V. Nguyen· Asian Journal of Research in...· 0 citations
The Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP) is proposed, enhancing both the robustness and granularity of vision-language alignment and underscore RoFLIP’s compositional reasoning and generalization abilities.
Yiwei Sun, Chuanbin Liu, Shancheng Fang et al.· International Journal of Com...· 0 citations
Multimodal Large Language Models (MLLMs) have achieved remarkable progress in Visual Question Answering (VQA), yet they continue to struggle with questions requiring precise spatial reasoning and fine-grained visual understanding. These limitations often manifest as object, attribute, and spatial hallucinations, where models generate confident but visually unsupported responses due to insufficient region-level and fine-grained visual grounding. To address this challenge, we propose ReVA, a region-aware VQA model that employs a frozen CLIP ViT-L/14 Vision Transformer (ViT) and a Qwen2.5-7B-Instruct large language model (LLM) connected through a dual bridge that aligns both whole-image and region-level representations with the LLM's embedding space. The image bridge maps final transformer block features into image tokens. The region bridge maps cropped features from enriched intermediate features across ViT blocks so early texture and later object cues are more evident, into K region tokens for every bounding box. ReVA uses a detector stack that supplies automatic zero-shot bounding boxes that are both question-agnostic and question-dependent, using RAM++ (Recognize Anything Model), spaCy, and Grounding DINO. The image tokens and region tokens are concatenated as an LLM prompt prefix to jointly encode scene-level context and fine-grained regional evidence when answering questions. Evaluated on VQAv2, MMBench, POPE, and SEED-Bench, ReVA achieves 82.85% mean F1 on POPE, compared with 81.14% for an image-token baseline without region tokens. These results demonstrate that explicit region-aware visual representations reduce object hallucination and improve the factual grounding of MLLMs.
The AP-LCA approach introduces a novel local cross-modal alignment strategy that utilizes image cropping and similarity-based semantic contribution assessment to precisely map fine-grained descriptions to relevant local image regions.
Si-Ying Wu, Song Wu· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
