Skip to content
Log in
Sponsored AI Networking Cookbook: Practical recipes for AI-assisted network automation and development AI Networking Cookbook: Practical recipes for AI-assisted network automation and development

Similar papers

Conference Jul 2026

Improving Trustworthiness in Visual Question Answering Via Question-Conditioned Cross-Modal Verification

Visual Question Answering (VQA) is a challenging cross-disciplinary task that combine natural language processing and computer vision together. VQA answer natural language questions based on images. Current VQA systems prone to generate plausible but incorrect responses without indicating uncertainty. To address this limitation, we propose a training-free cross-modal answer verification framework. This framework based on question-conditioned CLIP scoring as a post-hoc reliability estimator for vision-language models. The technique improves the ability to distinguish between right and wrong short-form VQA answers by evaluating candidate solutions jointly with the original question using type-aware distractor pools. Experiments on the VQAv2 validation set evaluate raw accuracy, verified accuracy, coverage, and accuracy gain under threshold-based selective prediction. Results shows consistent reliability improvements across diverse model architectures with gains ranging from +1.13 to +32.98 percentage points depending on coverage and model characteristics. The proposed framework is model-agnostic, requires no retraining or architectural modification, and improves the trustworthiness of multimodal systems by selectively filtering unreliable predictions. These findings show that question-conditioned CLIP verification provides an effective and scalable reliability layer for VQA systems.

Prakhar Shukla, Ankit Kumar, Pulkit Singh et al. · 0 citations
Sep 2026

Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domain

Multimodal large language models (MLLMs) have extended the capability of large language models (LLMs) to process more contextual multimodal information, showing remarkable progress in diverse realistic multimodal applications. Despite their strong perception and reasoning abilities, recent studies reveal that MLLMs remain highly vulnerable to adversarial inputs, especially those targeting visual components. However, existing attacks mainly focus on global perturbations, lacking an understanding of how MLLMs internally interpret visual structures. In this paper, we make the attempt to investigate the intrinsic focus of MLLMs in the frequency domain and discover that their predictions are particularly sensitive to phase information, which encodes essential structural and semantic cues. Based on this observation, we propose a novel phase-aware adversarial attack framework that explicitly restricts adversarial perturbations to structure-relevant phase regions to suppress the MLLMs'focus for effective and imperceptible attacks. To further amplify the structural influence, we also introduce an auxiliary adversarial prompt learning module to guide multimodal misalignment around phase-sensitive regions, misleading the MLLM's attention toward targeted structural patterns. Extensive experiments on multiple representative MLLM models and datasets demonstrate the superior effectiveness of our method compared to existing attacks.

Daizong Liu, Junhao Dong, Zhi-Yuan Ma et al. · 0 citations
Preprint Aug 2026

Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement

FISA is proposed, a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases that generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion.

Chun-Yang Jiang, Pingping Zhang, Yuzhi Zhao et al. · 0 citations
Aug 2026

Cross-modal alignment enhancement for lightweight large vision language models

A Low-Complexity Cross-Modal Alignment via Projection (LCAP) network is proposed, which introduces Projective Token Compression (PTC), which leverages Mish activation and adaptive average pooling to reduce feature redundancy while enhancing discriminative information, and Positional Spatial Enhancement (PSE), which explicitly injects positional cues into the compressed representations and strengthens spatial structure.

Yuchen Sha, Lingli Wan, Ge Yang et al. · 0 citations
#small language model Open access Aug 2026

Training-free counterfactual hallucination mitigation method for large vision-language models

This work proposes CounterfactualLVLM, a training-free and plug-and-play framework that mitigates object hallucinations via small-model-assisted counterfactual reasoning and highlights the power of counterfactual guidance as a simple yet effective paradigm for enhancing factual grounding in LVLM-based multi-modal reasoning.

Xilin Li, Boyue Wang, Xiao-Qian Ju et al. · 0 citations
Preprint Aug 2026

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

Existing research on object hallucination in multimodal large language models (MLLMs) predominantly attributes the problem to language priors such as over-reliance on textual co-occurrence statistics. We challenge this view by presenting quantitative evidence for a complementary, under-explored cause: visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings. Through cosine similarity analysis and Smooth Grad-CAM entropy measurements, we show that hallucinated samples exhibit systematically lower image-text similarity (average 0.158 vs. -0.122) and inverted attention patterns, where attention is dispersed when the target object is present but wrongly concentrated when it is absent. Guided by this diagnosis, we propose Adversarial Contrastive Fine-Tuning (ACFT). ACFT uses an Adversarial Hallucination Attribute Flipping (AHAF) procedure, involving minimal, targeted adversarial perturbations that flip an image's hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine-tuning. AHAF simultaneously serves as a diagnostic probe, revealing that MLLM visual representations lie dangerously close to hallucination decision boundaries. Requiring only 0.9% of the COCO dataset and adding zero inference overhead, ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL. Code is available at https://github.com/zxp555/ACFT_MM

Peiyang Xu, Xiaopei Zhu, Jun Zhu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.