EchoVision: A Decoupled Audio-Visual Retrieval Framework for Video Question Answering

Abstract

Multimodal video question answering (VideoQA) remains largely vision-centric, underutilizing auditory signals that are essential for real-world understanding. We propose EchoVision, a retrieval-based audio-visual framework that abandons explicit cross-modal fusion in favor of a decoupled dual-pipeline design with implicit alignment. Independent audio and visual streams are processed using frozen pretrained models-Whisper for speech transcription, PANNs-CNN14 for audio event detection, and keyframe-based captioning with temporal reasoning traces for visual understanding and unified through a shared CLIP embedding space. We introduce a unified multimodal evidence representation that integrates transcripts, sound events, captions, and reasoning traces into a single vector-indexed knowledge base. At inference, query-conditioned retrieval selects top- $k$ evidence, and a large language model generates answers via late fusion, enabling implicit Audio-Visual Segment Matching (AVSM) without any learned fusion or alignment parameters. Experiments on ActivityNet-QA, MovieChat1K, and the OfficeLab benchmark (490 QA pairs), newly curated by us to target audio-dependent scenarios, demonstrate consistent gains, with Answer Relevancy improving from 0.22 to 0.46 and Context Recall from 0.10 to 0.16.