Skip to content
#small language model Open access

Counterfactual Reasoning for Robust Visual Question Answering

Sep 2026 · Procedia Computer Science · Vol abs/2609.16567 · 0 citations · 23 references
Computer Science

TL;DR

A novel training framework that enhances counterfactual contrastive learning for VQA and introduces a three-stage curriculum for stable multi-objective optimization, an enhanced Batch-Contrastive loss for more discriminative feature learning and two novel regularizers.

Abstract

Modern Visual Question Answering (VQA) models often exploit spurious correlations in training data, leading to poor out-of-distribution (OOD) generalization due to language bias. Although counterfactual learning has shown promise, existing methods can be improved to better guide attention toward causal evidence and strengthen feature discrimination. To address this, we propose a novel training framework that enhances counterfactual contrastive learning for VQA. Our framework introduces three key contributions: (1) a three-stage curriculum for stable multi-objective optimization, (2) an enhanced Batch-Contrastive loss for more discriminative feature learning, and (3) two novel regularizers, Answer-Contrastive (AC) loss to refine the prediction space and Gradient-Discrepancy (GD) loss to enforce causal visual grounding. Our model achieves a competitive accuracy of 61.64% on the bias-sensitive VQA-CP v2 benchmark while maintaining 62.80% on the standard VQA v2 dataset, yielding a small generalization gap of 1.16%. This demonstrates a strong balance between OOD robustness and in-distribution performance.

Read PDF

Similar papers

#small language model Open access Aug 2026

Training-free counterfactual hallucination mitigation method for large vision-language models

This work proposes CounterfactualLVLM, a training-free and plug-and-play framework that mitigates object hallucinations via small-model-assisted counterfactual reasoning and highlights the power of counterfactual guidance as a simple yet effective paradigm for enhancing factual grounding in LVLM-based multi-modal reaso...

Xilin Li, Bo-Yue Wang, Xiao-Qian Ju et al. · 0 citations
#artificial intelligence Preprint Aug 2026

Counterfactual Contrastive Analysis

Visual Counterfactual Explanations (VCEs) aim to explain image classifiers by generating minimally edited and realistic versions of an input image that change the classifier's prediction. Existing VCE methods are inherently classifier-dependent and therefore susceptible to classifier biases and failure modes, such as s...

Yunlong He, Pietro Gori · 0 citations
#artificial intelligence Book Open access Sep 2026

When Helpful Text Hurts: Option-Redirecting Bias in Vision–Language Models

In tri-modal visual question answering (VQA), auxiliary text is commonly used to complement visual and textual inputs, yet its reliability is often uncontrolled. While prior work studies modality conflicts in general, it remains unclear how different types of unreliable auxiliary text affect answer selection under fixe...

Tam Le Thi Thanh, Van Tran Hoang, Hong-Hanh Nguyen-Le et al. · 0 citations
Preprint Aug 2026

FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding

This paper presents FADE, an effective training framework for counterfactual discovery and explanation that is built on an evidence-first, two-stage training paradigm, and demonstrates remarkable robustness when transitioning from constrained MCQs to unconstrained OQA and captioning.

Fufangchen Zhao, Jin-Hu Fu, Jiachen Lei et al. · 0 citations

Question-Specific Knowledge Graphs for Efficient Visual Reasoning

Recent work in visual question answering has shown that vision-language models can exhibit strong reasoning capabilities by translating visual inputs into textual representations. The effectiveness of this translation depends on how well visual details are retained; models need to surface and align both explicit and im...

Ting-Chih Chen, Emile van Krieken, Shu-Jian Yu et al. · 0 citations
Preprint Sep 2026

Targeted Visual Counterfactual Explanations for Contrastive Vision-Language Model

Current explanation methods for contrastive vision-language models such as CLIP mainly identify important regions without showing how to change the input in order to get a target prediction. We introduce Mask-guided Adaptive Counterfactual Explanations (MACE), a targeted visual counterfactual method designed specifical...

Van Bach Nguyen, Jörg Schlötterer, Christin Seifer · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.