Robust Audio-Visual Question Answering with Missing Modality in Training and Testing.

Abstract

Audio-Visual Question Answering (AVQA) requires reasoning over temporally evolving audio and visual signals to answer natural-language questions about dynamic scenes. Most existing methods assume that both modalities are available during training and testing. In practice, however, an audio or visual stream may be unavailable because of passive signal loss, such as hardware or transmission failures, or intentional removal, such as withholding visual information for privacy. We formulate Training-time Modality-Missing AVQA (TM-AVQA), a setting in which modality-complete, audio-missing, and visual-missing samples may occur during both training and testing. To address this new setting, we develop an AVQA-specific two-stage framework that adapts established cross-modal reconstruction and dependency-modeling principles to the supervision constraints of TM-AVQA. In Stage-I, a reconstruction network infers task-oriented feature-level representations of the missing modality from the available modality and question text. Its Dense Temporal-scale Reconstruction (DTR) module aggregates complementary information across multiple temporal granularities, while its Multimodal Dependency Modeling (MDM) module captures dependencies among the audio, visual, and question modalities. To supervise reconstruction when the target-modality features are unavailable, we adapt two complementary learning objectives to TM-AVQA: Cross-Modal Relation-based Contrastive Learning (CMR-CL), which exploits within-video and cross-video multimodal relations, and Cross-Sample Relation-based Pseudo-label Learning (CSR-PL), which constructs feature-level pseudo targets from relevant modality-complete samples. In Stage-II, the frozen reconstruction network is combined with existing AVQA backbones for answer prediction. We construct controlled TM-AVQA variants of MUSIC-AVQA, MUSIC-AVQA-R, and AVQA datasets by deleting one modality from selected samples. Experiments across multiple AVQA backbones, missing-modality conditions, and missing rates show consistent and robust improvements over the evaluated AVQA and missing-modality baselines. Additional ablations and analyses examine the contribution of the proposed task-specific designs, reconstruction quality, efficiency, and transfer to audio-visual recognition task.