Official short-answer VQA scores should be accompanied by semantic audits and answer-type diagnostics to remain interpretable, because instability is structured by answer type: extractive and multi-span answers are far more evaluator-sensitive than scalar answers.
Abstract
Short-answer VQA benchmarks conflate two distinct quantities: whether a model's answer is semantically correct, and whether that answer matches the surface form expected by the automatic evaluator. We study this conflation across six vision--language models and six benchmarks, using a human-validated semantic judge (97.6% precision) to audit over 37k official errors. A second text-only judge reproduces the same benchmark-level false-negative pattern, showing that the effect is not an artifact of a single audit model. On text-rich benchmarks, up to half of these errors are semantically acceptable answers penalized purely for surface-form mismatch. This instability is structured by answer type: extractive and multi-span answers are far more evaluator-sensitive than scalar answers. Benign prompt and context rewrites further destabilize official outcomes, flipping item-level correctness at substantial rates without changing the underlying task. A deterministic CPU-only contract repair confirms that the undercount is partially recoverable. These findings imply that official short-answer VQA scores should be accompanied by semantic audits and answer-type diagnostics to remain interpretable.
Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks, free-form answers may be correct in many surface forms and may fail in qualitatively different ways, including incompleteness, contradiction, overgeneration, and endorsement of false premises. Existing judgment-based and similarity-based metrics often collapse these distinctions. We address this gap with three reusable contributions. First, we introduce a semantic correctness taxonomy that assigns open-ended answers to eight ordered classes, separating verbose-but-correct answers from those contaminated by hallucinated content. Second, we release CAP-Correctness, an 8.8k-example benchmark spanning widely used QA datasets, and CAP-Statements, an 11k-example dataset for converting question-answer pairs into declarative statements for natural language inference (NLI) training and statement-based evaluation. Third, we introduce CAP (Context-Aware Precision), a reference-based metric that scores question-conditioned statements using bidirectional NLI. Under a monotonicity protocol testing whether metrics respect the taxonomy's intended ordering, CAP outperforms established baselines.
Elitsa Yotkova, Violeta Kastreva, Petar Velkov et al.· 0 citations
LLM judges are increasingly being used to evaluate open-ended model responses, often in no-reference settings where a ground-truth answer is unavailable. However, can they reliably assess in such evaluation setups? We explore this question in this paper through a two stage pipeline with a) calibration experiments that assess the judge model's knowledge of the task it is evaluating, and b) sensitivity experiments that assess how the judge model's performance is impacted by the presence and positioning of the reference answer in the prompt. Across experiments covering three languages, we show that the judge models we evaluated tend to over-credit incorrect answers in the absence of a reference answer, and adding reference answer information to the prompt flips the judge model's correct/incorrect decisions by as much as 85% in some experimental settings. Comparison with a subset of human annotations shows that these reference-driven changes generally align with human judgments. Our results emphasize the need for calibrating the LLM judges with a sample with reference-aware evaluation before using them in reference-free setups reliably, and our methodology provides a blueprint for researchers and practitioners in doing such calibration of LLM judges for other tasks.
Evaluating open-ended large language model responses remains difficult because response quality depends not only on factual correctness and task completion, but also on subjective and scenario-dependent user experience factors. Existing benchmarks and automatic evaluators are effective for coarse-grained assessment, yet often provide limited insight into how different types of quality failures affect user-perceived response quality. To address this gap, we propose AutoQABench, an initial three-level user experience benchmark for automated evaluation of open-ended LLM responses. The benchmark decomposes response quality into three progressively organized levels: basic acceptability constraints, scenario-specific task effectiveness, and preference-sensitive experiential quality. Based on this design, we construct a dataset covering representative scenarios, including summarization, elaboration, emotional support, and responses to misleading premises, together with expert-defined evaluation standards. We also develop an LLM-based modular evaluator that performs stepwise assessment and generates both intermediate judgments and a final rating. Experimental results show that AutoQABench achieves good agreement with expert judgments and supports analysis at the module, scenario, and final-grade levels. Rather than replacing human evaluation, AutoQABench is intended as a scalable auxiliary evaluator that helps identify overlooked risks, task-completion failures, and quality differences in open-ended LLM interactions.
Tianyou Wang, Fei Yuan, Chia-Ju Miao et al.· International Conference on...· 0 citations
Small language models can grade open-ended examination answers as reliably as substantially more expensive models when they grade against an explicit rubric. We test this claim as the design principle behind any-to-bench: a frontier model reads source documents once, at ingestion, to extract each question and its rubric; lower-cost models then perform all repeated grading work. We evaluate six cost-efficient model configurations from two model families at three reasoning-effort levels. Each configuration answers 24 open-ended examination questions, and each also grades every answer sheet three times, yielding 3,456 per-question grades. Scores depend overwhelmingly on the answer being graded: answer identity explains 95.6% of score variance, whereas judge identity explains only 0.2%. Raising a writer's reasoning effort moves earned scores by as much as 0.143 of full marks, while raising a judge's reasoning effort moves assigned scores by at most 0.006. Six frontier-tier judges, added as a check, reproduce these scores and are no more reliable as a panel. Two ablations then decompose the rubric on the same questions and answers. Removing its criteria and levels while keeping the official answer changes nothing measurable. Removing the official answer as well collapses reliability (ICC 0.888 to 0.628), inflates scores, and makes judge reasoning effort matter again. The rubric is what decouples grading from judge intelligence, and within the rubric the official answer does nearly all the work. We find no evidence of length preference or same-family preference under rubric-anchored grading.
CROWN-QA is introduced, comprising CROWN-Synth, a controlled paired core that fixes the question and observed facts while varying only query-relative coverage, and CROWN-Real, a real-document contrast-set evaluation with controlled coverage variants.
Byoungjae Min, Kennedy Edemacu, Sae-Hong Cho et al.· 0 citations
Three novel metric models are proposed: a prompt-based strategy utilizing Large Language Models to assess answers, an approach that adapts precision and recall concepts by segmenting answers into discrete information units, and a regression model trained on synthetic data to predict completeness and relevance scores.
E. Cortes, Renata Vieira, D. Barone· Language Resources and Evalu...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
