ST-Evidence is introduced, the first human-verified benchmark for both discriminative and generative pixel-level grounding, and scalable, automated generation pipelines are developed to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding.
Abstract
Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.
We study a critical yet overlooked failure mode in Grounded Video Question Answering: question-invariant grounding, where models predict nearly identical temporal segments for different questions about the same video. We trace this behavior to two structural limitations in prior common designs: (i) modality isolation that fixes video representations before they receive question semantics, and (ii) weak question injection inside the grounding module. To address this, we propose GroundFormer, which conditions video features on question intent before localization via learnable communication tokens that mediate directed visuo-lingual interaction. On top of the question-conditioned features, a factorized MIL cross-attention couples answer selection with temporal evidence under candidate-level supervision, while Gaussian smoothing converts peaked attention into temporally coherent segments. We further introduce a hierarchical multi-modal contrastive loss that aligns video, question, and answer embeddings across a two-pass training pipeline. GroundFormer achieves state-of-the-art grounded VideoQA performance on NExT-GQA and STAR, substantially improving question-discriminative temporal grounding.
Jinhwan Seo, Kyu-Tae Han, Jumin Lee et al.· 0 citations
Experimental results demonstrate that S2V can consistently improve LVU performance across multiple LVU benchmarks, outperforming both general MLLMs and reasoning-based methods not only in LVU accuracy but also in training and inference efficiency.
Beibei Zhang, Chao Xu, Jun Lan et al.· 0 citations
Multimodal video question answering (VideoQA) remains largely vision-centric, underutilizing auditory signals that are essential for real-world understanding. We propose EchoVision, a retrieval-based audio-visual framework that abandons explicit cross-modal fusion in favor of a decoupled dual-pipeline design with implicit alignment. Independent audio and visual streams are processed using frozen pretrained models-Whisper for speech transcription, PANNs-CNN14 for audio event detection, and keyframe-based captioning with temporal reasoning traces for visual understanding and unified through a shared CLIP embedding space. We introduce a unified multimodal evidence representation that integrates transcripts, sound events, captions, and reasoning traces into a single vector-indexed knowledge base. At inference, query-conditioned retrieval selects top- $k$ evidence, and a large language model generates answers via late fusion, enabling implicit Audio-Visual Segment Matching (AVSM) without any learned fusion or alignment parameters. Experiments on ActivityNet-QA, MovieChat1K, and the OfficeLab benchmark (490 QA pairs), newly curated by us to target audio-dependent scenarios, demonstrate consistent gains, with Answer Relevancy improving from 0.22 to 0.46 and Context Recall from 0.10 to 0.16.
Subhajit Sarkar, Joyita Chakraborty, Sagnik Nandi et al.· International Conference on...· 0 citations
While LVLMs rapidly improve, long-video question answering still remains challenging: relevant evidence is sparse, and question-relevant context often fails to provide cues that discriminate the correct answer from plausible alternatives. Diagnostic analysis on a manually annotated subset of MMR-V shows that prior agentic systems substantially improve cue retrieval over direct VLM inference yet fail to achieve a corresponding gain in answer accuracy, indicating that the bottleneck lies in option-discriminative evidence rather than topical relevance alone. We propose PACE (Progressive Acquisition of Critical Evidence), a factor-guided framework for long-video evidence acquisition. PACE proceeds in two stages: it first indexes clip-level descriptions guided by question-derived factors without observing the candidate answers; it then uses the candidate answers to derive contrastive cues and queries the index for verification. On MMR-V with the open-source Qwen3-VL backbone, PACE achieves 42.6% accuracy, outperforming direct inference and prior agentic baselines including Deep Video Discovery (DVD). On the same diagnostic subset, PACE recovers 66.9% of the annotated cues, providing empirical evidence that its gains are associated with improved evidence recovery rather than stronger answer-side priors alone. Consistent gains over DVD on LVBench, Video-MME, EgoSchema, and LongVideoBench suggest that option-aware evidence acquisition transfers beyond MMR-V. Code is available at https://github.com/HKUST-KnowComp/PACE.
Baixuan Xu, Yinyui Xu, Tianshi ZHENG et al.· 0 citations
Long-video question answering (QA) forces multimodal large language models (MLLMs) to work within a tight frame budget, so the choice of frames largely decides whether a question can be answered at all. The standard recipe scores every frame against the question with a pretrained image–text matching (ITM) model and keeps the top scorers. A fundamental mismatch underlies this recipe: ITM models are trained on short, concrete visual descriptions, while QA questions are interrogative and often involve abstract terms. Scored against the question alone, the ITM yields a near-random signal whenever the question is not a direct image–text match, such as one asking for the temporal order of scenes. In our LongVideoBench diagnostic analysis, the score collapses even on benchmark-provided answer-relevant frames, with more than half falling into a near-zero region —not because the encoder is faulty, but because it behaves exactly as it was trained to. We argue that this format mismatch should be absorbed at the two ends of the pipeline while the encoder itself stays frozen. On the input side, a type-conditioned routed pipeline reformulates each question into a single ITM-aligned description by selectively applying grounding, decomposition, and constrained synthesis. The ITM therefore receives exactly one description per frame, preserving the per-frame matching cost of a standard single-query baseline. On the output side, because the score distribution remains polarized and answer frames are scattered in time, we replace top- $K$ selection with a parameter-free Rosin threshold followed by a temporal maximal-marginal-relevance (MMR) step that uses frame positions alone. Across three benchmarks (LongVideoBench, Video-MME, MLVU) and and four backbones (Qwen2-VL, Qwen2.5-VL, LLaVA-OneVision, LLaVA-Video), the resulting training-free pipeline, RECAST, consistently outperforms recent frame-selection baselines without modifying the ITM encoder.
S. Han, Thang Vu, Junyeong Kim· IEEE Access· 0 citations
Multimodal Large Language Models (MLLMs) have achieved remarkable progress in Visual Question Answering (VQA), yet they continue to struggle with questions requiring precise spatial reasoning and fine-grained visual understanding. These limitations often manifest as object, attribute, and spatial hallucinations, where models generate confident but visually unsupported responses due to insufficient region-level and fine-grained visual grounding. To address this challenge, we propose ReVA, a region-aware VQA model that employs a frozen CLIP ViT-L/14 Vision Transformer (ViT) and a Qwen2.5-7B-Instruct large language model (LLM) connected through a dual bridge that aligns both whole-image and region-level representations with the LLM's embedding space. The image bridge maps final transformer block features into image tokens. The region bridge maps cropped features from enriched intermediate features across ViT blocks so early texture and later object cues are more evident, into K region tokens for every bounding box. ReVA uses a detector stack that supplies automatic zero-shot bounding boxes that are both question-agnostic and question-dependent, using RAM++ (Recognize Anything Model), spaCy, and Grounding DINO. The image tokens and region tokens are concatenated as an LLM prompt prefix to jointly encode scene-level context and fine-grained regional evidence when answering questions. Evaluated on VQAv2, MMBench, POPE, and SEED-Bench, ReVA achieves 82.85% mean F1 on POPE, compared with 81.14% for an image-token baseline without region tokens. These results demonstrate that explicit region-aware visual representations reduce object hallucination and improve the factual grounding of MLLMs.
A. Senthil· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
