VSeek is introduced, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process and proposes a novel neuro-symbolic approach that bridges open-ended natural language with discrete visual verification.
Abstract
We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process. VSeek utilizes a natural language-driven search to identify relevant context within long videos and is post-trained with reinforcement learning (RL) to jointly formulate targeted search queries and reason over retrieved clips for LVQA. While RL post-training has revolutionized reasoning in symbolic domains such as mathematics and code, its application to long-video understanding remains hindered by a lack of verified rewards. To ensure that the retrieved context is relevant, we propose a novel neuro-symbolic approach that bridges open-ended natural language with discrete visual verification. Specifically, complex user queries are compiled into formal temporal logic specifications for systematically decomposing natural language questions into a definitive checklist of required atomic visual primitives, such as key objects and activities, along with their temporal ordering. These systematically derived grounding events provide the critical feedback signal for RL post-training, enabling dense, verifiable rewards based on the successful retrieval of these specific visual elements rather than relying entirely on outcome-only answer accuracy. By explicitly optimizing for this verifiable evidence-seeking behavior, VSeek improves Pass@1 scores by up to 8% and Pass@4 scores by 15% on long-video understanding benchmarks compared to base models. We open-source our code at https://utaustin-swarmlab.github.io/VSeek.
ReVEAL consistently outperforms both closed-source and open-source state-of-the-art methods across extensive experiments and shows that explicitly verifying evidence sufficiency, rather than stopping at semantic relevance, retrieves the decisive clues that prior methods miss and yields more reliable long-video reasoning.
C.J. Yan, Yang Zhou, Meixing Shi et al.· 0 citations
Long-video question answering (QA) forces multimodal large language models (MLLMs) to work within a tight frame budget, so the choice of frames largely decides whether a question can be answered at all. The standard recipe scores every frame against the question with a pretrained image–text matching (ITM) model and keeps the top scorers. A fundamental mismatch underlies this recipe: ITM models are trained on short, concrete visual descriptions, while QA questions are interrogative and often involve abstract terms. Scored against the question alone, the ITM yields a near-random signal whenever the question is not a direct image–text match, such as one asking for the temporal order of scenes. In our LongVideoBench diagnostic analysis, the score collapses even on benchmark-provided answer-relevant frames, with more than half falling into a near-zero region —not because the encoder is faulty, but because it behaves exactly as it was trained to. We argue that this format mismatch should be absorbed at the two ends of the pipeline while the encoder itself stays frozen. On the input side, a type-conditioned routed pipeline reformulates each question into a single ITM-aligned description by selectively applying grounding, decomposition, and constrained synthesis. The ITM therefore receives exactly one description per frame, preserving the per-frame matching cost of a standard single-query baseline. On the output side, because the score distribution remains polarized and answer frames are scattered in time, we replace top- $K$ selection with a parameter-free Rosin threshold followed by a temporal maximal-marginal-relevance (MMR) step that uses frame positions alone. Across three benchmarks (LongVideoBench, Video-MME, MLVU) and and four backbones (Qwen2-VL, Qwen2.5-VL, LLaVA-OneVision, LLaVA-Video), the resulting training-free pipeline, RECAST, consistently outperforms recent frame-selection baselines without modifying the ITM encoder.
S. Han, Thang Vu, Junyeong Kim· IEEE Access· 0 citations
Document question answering over scanned pages requires two coupled abilities: (i) canonicalizing complex layouts into a faithful textual structure, and (ii) selecting and reasoning over query-relevant evidence from that structure. Most existing pipelines decouple OCR from retrieval-augmented reasoning and optimize OCR for global reconstruction, which often misaligns with evidence needs and causes brittle grounding in multi-page settings. We propose UniDocVLM, an end-to-end framework that unifies OCR and visual RAG within a single vision-language model: the model first generates a structured parse of retrieved pages, then activates question-relevant evidence from the parse to support grounded reasoning and answering. To train UniDocVLM under heterogeneous supervision, we introduce a unified JR-GRPO reinforcement learning recipe with lightweight, verifiable rewards, including format, layout-aware OCR, evidence-consistency, and answer-correctness signals, and route them to the corresponding parts of the output to improve credit assignment and reduce interference. Experiments on multi-page document QA benchmarks show that UniDocVLM yields more reliable evidence grounding and improves downstream accuracy under complex layouts.
Zong-Sheng Cao, Anran Liu, Jun Xie et al.· Proceedings of the 32nd ACM...· 0 citations
Large Vision-Language Models (LVLMs) have become essential for advancing the integration of visual and linguistic information. While existing benchmarks have laid a solid foundation for evaluation, they are often static, resource-intensive to build, and limited in adaptability. In comparison, automatic evaluation has shown promise in the textual domain, but the visual modality remains far less explored. To advance this frontier, in this work, we introduce AutoDavis, a first-of-its-kind automatic and dynamic evaluation protocol that enables on-demand benchmarking of LVLMs across specific capability dimensions. AutoDavis leverages text-to-image models to generate relevant image samples and then utilizes LVLMs to orchestrate visual question-answering (VQA) tasks, completing the evaluation process efficiently and flexibly. To ensure data diversity, our framework employs a hierarchical aspect-driven generation process enhanced with semantic graph-based constraints. To safeguard reliability, the framework incorporates a self-validation mechanism to detect and correct errors, along with an error-driven adjustment module to mitigate potential bias. Through an extensive evaluation of 11 popular LVLMs across five demanded user inputs (i.e., evaluation capabilities), the framework shows effectiveness and reliability, offering a new paradigm for dynamic benchmarking of multimodal intelligence. View website for code and data.
Han Bao, Yue Huang, Yanbo Wang et al.· Proceedings of the 32nd ACM...· 0 citations
Clue-OPSD, a clue-privileged on-policy self-distillation framework for long-video understanding that uses clue intervals as privileged supervision without relying on ground-truth answer labels, while requiring no clue annotations or additional modules at inference time is introduced.
Kaishen Wang, Dong-Di Zhao, Yijun Liang et al.· 0 citations
Perception-RFT is introduced, a training framework that applies Group Relative Policy Optimization (GRPO) to multimodal document QA, bypassing intermediate reasoning tokens to directly align visual features with structured grounding outputs, and a reasoning variant under identical reward settings is constructed.
M. HarikrishnanP., G. Vignesh, Ganesh Parab et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
