ReDA: discrepancy guided multimodal fusion for audio-visual deception detection

Abstract

Deception detection often manifests as inconsistency between visual and audio cues, in which the speaker’s visual expression may conflict with their vocal pitch. Previous work has typically focused on complex attention mechanisms or concatenation-based fusion methods, assuming that both channels have identical reliability levels in all scenarios. As a result, they are highly susceptible to the deterioration of a particular modality caused by production artifacts, background noise, and facial occlusion. In this paper, we propose ReDA, a Receptive-field Detail-preserving and Discrepancy-Aware network for audio-visual deception detection. In the visual stream, we design a new Receptive-field Detail-Preserving (ReDP) block by integrating a dilated strided convolution pathway with two pooled pathways, thereby increasing the receptive field while retaining detailed facial texture features. For processing the acoustic channel, we adopt a self-supervised speech model that utilizes audio features, with masked mean pooling to make it robust to noise or compression during transmission. The key to our approach is a discrepancy-aware dynamic fusion module, which treats cross-modality inconsistency as an explicit indicator of deception. Besides capturing the agreement and disagreement between the two modalities, the module creates an interaction representation from which a sample-specific reliability level of each modality can be inferred and dynamically adjusted. An entropy-based regularization technique is further applied to promote equal contribution from both channels during training. We demonstrate the effectiveness of the ReDA framework through experiments, reporting state-of-the-art results.