Results show that process priors are most useful when aligned with the scene's specific decision boundary, motivating boundary-aware prior selection for process-grounded visual reasoning.
Abstract
Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they have already identified correctly. We distinguish perceptual failure, where relevant evidence is not recognized, from process failure, where recognized evidence fails to constrain the final decision. We introduce VPAC-Bench, a benchmark spanning nine real-image process families, with each image annotated by its current activity stage and nearby stage transition. We also propose State-Relevance-Target (SRT), a family of structured process-prior interventions that requires models to connect visible evidence to the relevant process state before answering. Across multiple VLMs, process failure is widespread: models that correctly enumerate visual candidates still over-commit to a single answer in more than 95% of ambiguous cases. An explicit process-structured intervention reduces this rate to below 13% without degrading performance on unambiguous cases. However, the transfer of process priors is model-dependent, and generic SRT does not consistently outperform strong chain-of-thought baselines. When the relevant stage transition is known, boundary-aligned SRT substantially outperforms generic process prompting and all tested chain-of-thought baselines across assembly, physical state transition, navigation and traffic, and object-use affordance tasks. These results show that process priors are most useful when aligned with the scene's specific decision boundary, motivating boundary-aware prior selection for process-grounded visual reasoning.
Vision-language models (VLMs) answer visual questions by combining visual information extraction with downstream problem solving. We investigate a fundamental question: Does an incorrect answer necessarily reflect a failure in visual extraction or problem solving? A model may succeed at both abilities when tested separ...
Zi-Heng Wang, Ming-Xuan Xie, Yi-Lin Liu et al.· arXiv.org· 0 citations
This work proposes Counterfactual Evidence Disentanglement (CED), a training-time evidence audit for VLM grounding, which outperforms prior RL-based post-training methods, with targeted analyses verifying its object-centric signal.
Haojie Huang, Xin-Lei Yu, Cheng-Ming Xu et al.· 3 citations
A small detector is trained to predict the internal development of answer readiness and stop reasoning at that point, connecting the internal development of answer readiness to a practical rule for allocating reasoning computation.
This paper shows that Vision-Language Models (VLMs) can internally distinguish when abstention is required, but fail to express it anyway, and introduces Penalized Epistemic Calibration Score (PECS), a new robust metric that requires models to both answer correctly when the outcome is knowable, and abstain when the out...
Fnu Pramono, J. Cai, Sourabh Kulkarni· 5 citations· ⚡1
The contrastive evidence gap is formalized, the per-token log-likelihood ratio that a model assigns to its own output when conditioned on a question-relevant region versus an irrelevant one, and it is shown that the gap is complementary to the model's own confidence.
Santiram Tiwari, N. Naik, Devbrat Pandey et al.· 0 citations
VSI is not a universal best abstention signal; it is a sample-intrinsic indicator of vision-ignoring failure, best used as a conditional ensemble component.
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.