When Verification Hurts: The Cost of Overriding Abstention in Two-Stage Web Agents

Abstract

Web agents built on large vision-language models (VLMs) increasingly adopt a two-stage design: a grounding stage proposes candidate elements on a page, and an action stage decides which element to operate on and how. A natural way to strengthen such agents is to insert a pre-action verifier that re-scores the grounded candidates before acting, echoing the gains that verification and self-refinement bring to language-model reasoning. We test this assumption on the Mind2Web benchmark and report a counter-intuitive result: a GLM-4.6V pre-action verifier does not help and in fact degrades performance, low-ering the action-level step success rate from 34.8% to 23.7% on our evalua-tion subset. Through a step-level analysis we attribute this degradation to two causes. First, the offline multiple-choice protocol has limited candidate cov-erage, as the gold element is absent from the candidate set in roughly 80% of steps, so most steps are unsolvable regardless of verification. Second, and more decisively, the verifier mis-ranks candidates on the solvable steps and discards the grounding stage's calibrated abstention on the unsolvable majori-ty, so it removes a safe default without improving accuracy: it wins 5 steps but loses 20. Guided by this diagnosis, we propose an abstention-aware veri-fier that intervenes only under sufficient candidate coverage and confidence. Our study cautions against transplanting verification into grounding pipelines and identifies calibrated abstention as a property worth preserving.