AutoQABench: A Three-Level UX Benchmark for Automated Evaluation of Open-Ended LLM Responses

A benchmark dataset with human validation for AI-assisted technical answer evaluation

Grading open-ended technical responses has been a longstanding issue in higher education. Despite advances in automated assessment, existing approaches often rely on holistic scoring, weakly validated annotations, and cognitive alignment, limiting pedagogical reliability and classroom adoption. To facilitate reliable and rubric-based automated evaluation in the Data Structures and Algorithms course, this study presents DSA-RubricEval, a pedagogically grounded and preliminarily reliability-assessed dataset. The dataset constitutes the primary contribution of this work, providing a structured benchmark aligned with Bloom’s taxonomy and validated through multi-rater Interclass Correlation Coefficient ICC analysis. The dataset consists of twelve expert-designed, Bloom-aligned, questions scored on multiple rubric dimensions using an ordinal scale. Five independent evaluators scored student responses, enabling rigorous validation of human judgement based on the Intraclass Correlation Coefficient (ICC) analysis. The results indicate good to excellent average-measure reliability across most rubric dimensions, justifying the use of aggregated human scores as aggregated reference labels for automated assessment. Automated scoring was explored as a proof of concept to demonstrate the applicability of the dataset for AI-assisted assessment and formulated as an ordinal, rubric-level prediction task and evaluated using pedagogically motivated agreement measures, showing high tolerance-based agreement with human judges. The proposed research identifies sources of assessor subjectivity and explores methods to mitigate them, while reducing the workload of grading as well as correlate with learning outcomes. Moreover, the proposed technique supports lower-order cognitive skills but is not best suited for higher-order cognitive tasks. Overall, this study introduces a preliminarily reliability-assessed, rubric-based benchmark dataset intended to support exploratory research on pedagogically meaningful AI-assisted assessment of open-ended technical responses.

Semantic anchoring with concise ideal answers outperforms unstructured full course materials as context for multi-LLM automated grading of open-ended questions

Large language models (LLMs) are increasingly used to grade open-ended student responses, yet the role of contextual input in this process remains poorly understood. This study compares three context conditions for multi-LLM automated grading: no context, full course materials, and instructor-defined ideal answers as semantic anchors. Using a dataset of 3,041 student responses (3,011 after common-support exclusions) to 50 open-ended questions from an undergraduate computer science course, we evaluated three base LLMs (DeepSeek, Qwen, Gemini) against grades derived from two independent blind instructor assessments. A factorial analysis based on the Aligned Rank Transform revealed significant main effects of model and condition, with a significant interaction. Ideal-answer anchoring significantly outperformed both alternatives in absolute grading error, while providing full course materials significantly worsened accuracy relative to the no-context baseline. The anchored condition achieved the lowest mean absolute error (1.268), the highest correlation with instructor grades ( r  = 0.801), and the lowest inter-model disagreement (median SD = 0.864), at a per-response cost comparable to the no-context baseline (EUR 0.00119 vs. 0.00113) and 4.2 times cheaper than the full-materials condition (EUR 0.00501). The absolute accuracy gain over the no-context baseline is small (≈ 0.08 points on a 0–10 scale; marginal R 2 = 0.013); its practical value lies in the convergence of accuracy, inter-model agreement, feedback-quality and cost improvements and in avoiding the accuracy loss caused by unstructured full course materials. A complementary analysis of 27,099 feedback instances, validated against a human gold standard (κ = 0.898), showed that out-of-scope feedback decreased by 25.4% under semantic anchoring, with a logistic regression revealing that this benefit was concentrated in two of the three evaluators. Evidence derives from a single course, institution, language (Spanish) and academic year, and from convergent-answer theoretical assessment; within this setting, concise instructor-defined ideal answers (rather than large volumes of unfiltered course material) yielded the most reliable grading and feedback.

EduFairBench: reproducible evaluation of large language models for educational assessment

Large language models (LLMs) are increasingly used to evaluate open-ended educational responses. However, their performance is often assessed using aggregate metrics that provide limited insight into prediction stability, uncertainty, error patterns, and feedback quality. This study presents EduFairBench, a reproducible evaluation protocol designed to characterize LLM behavior across short-answer assessment and automated essay scoring using open educational benchmarks. The protocol combines repeated inference, majority-vote consolidation, uncertainty estimation, error analysis, and structural evaluation of generated feedback within a unified experimental framework. Experiments were conducted on SciEntsBank, Beetle, and ASAP2, comprising 2,000 student responses and 10,000 independent LLM inferences. The results showed moderate predictive agreement with human assessment while revealing substantial differences between nominal and ordinal evaluation tasks. Repeated inference demonstrated high internal stability across benchmarks, although systematic errors remained in semantically adjacent categories, indicating that prediction consistency does not necessarily imply correctness. Feedback quality varied by task type, with longer textual contexts yielding more specific and pedagogically structured explanations. These findings demonstrate that evaluating educational LLMs requires complementary analyses beyond conventional performance metrics. EduFairBench provides a reproducible methodology for jointly analyzing predictive performance, robustness, uncertainty, and feedback quality, providing a comprehensive methodological framework for the rigorous evaluation of LLM-based educational assessment systems.

What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarks

Official short-answer VQA scores should be accompanied by semantic audits and answer-type diagnostics to remain interpretable, because instability is structured by answer type: extractive and multi-span answers are far more evaluator-sensitive than scalar answers.

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks, free-form answers may be correct in many surface forms and may fail in qualitatively different ways, including incompleteness, contradiction, overgeneration, and endorsement of false premises. Existing judgment-based and similarity-based metrics often collapse these distinctions. We address this gap with three reusable contributions. First, we introduce a semantic correctness taxonomy that assigns open-ended answers to eight ordered classes, separating verbose-but-correct answers from those contaminated by hallucinated content. Second, we release CAP-Correctness, an 8.8k-example benchmark spanning widely used QA datasets, and CAP-Statements, an 11k-example dataset for converting question-answer pairs into declarative statements for natural language inference (NLI) training and statement-based evaluation. Third, we introduce CAP (Context-Aware Precision), a reference-based metric that scores question-conditioned statements using bidirectional NLI. Under a monotonicity protocol testing whether metrics respect the taxonomy's intended ordering, CAP outperforms established baselines.

Understanding the gap in evaluation frameworks and metrics for RAG-based question-answering system