Beyond accuracy: completeness and relevance metrics for evaluating the quality of long answers

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks, free-form answers may be correct in many surface forms and may fail in qualitatively different ways, including incompleteness, contradiction, overgeneration, and endorsement of false premises. Existing judgment-based and similarity-based metrics often collapse these distinctions. We address this gap with three reusable contributions. First, we introduce a semantic correctness taxonomy that assigns open-ended answers to eight ordered classes, separating verbose-but-correct answers from those contaminated by hallucinated content. Second, we release CAP-Correctness, an 8.8k-example benchmark spanning widely used QA datasets, and CAP-Statements, an 11k-example dataset for converting question-answer pairs into declarative statements for natural language inference (NLI) training and statement-based evaluation. Third, we introduce CAP (Context-Aware Precision), a reference-based metric that scores question-conditioned statements using bidirectional NLI. Under a monotonicity protocol testing whether metrics respect the taxonomy's intended ordering, CAP outperforms established baselines.

Understanding the gap in evaluation frameworks and metrics for RAG-based question-answering system

Design and Implementation of a Sentence-BERT-Driven Semantic Matching Model for Automated Evaluation of Theory Answers

Theory-based assessments remain one of the hardest grading challenges to automate. Students rarely express correct answers the way a model answer expects, and the keyword-matching tools built into most learning management systems penalise them for it, not because they are wrong, but because they phrased things differently. EvalAI addresses this by grading theory answers on meaning rather than wording, using Sentence-BERT to measure how semantically similar a student's response is to a lecturer-provided reference answer. Grading suggestions are surfaced to the lecturer, who retains full control over final marks. Evaluation against Mohler's short-answer grading benchmark yielded a Pearson correlation of 0.81 and a mean absolute error of 0.18, indicating that the system's suggestions align closely with human judgement. All functional test cases passed during system testing, and evaluators from both lecturer and student roles completed their tasks without assistance. The results show that semantic grading of this quality can be delivered within a deployable web application without specialised machine learning infrastructure, while keeping the lecturer firmly in the grading loop.

A dataset of rated conceptual arguments

Large language models have improved rapidly on tasks with verifiable answers, such as mathematics and programming. Much less is known about their ability to reason about what we call conceptual questions: questions for which no ground truth is realistically accessible and no widely accepted resolution methodology exists, but on which progress can still be made by debating arguments. Most philosophical questions are of this kind, as are central components of questions in AI safety, decision theory, and social choice. Our approach is based on the view that while bottom-line conclusions on such questions are hard to evaluate, individual contextualized arguments can be evaluated far more reliably. We therefore introduce a dataset of 951 argumentative critiques of 442 position texts, spanning topics from AI safety and decision theory to ethics and politics, with 1,458 ratings by six expert raters along dimensions including centrality, strength, correctness, and clarity. We propose two scoring functions and benchmark a range of models. Performance tracks general capability rankings.

LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering

By evaluating paper retrieval, evidence grounding, and answer accuracy separately, LitTraceQA provides a testbed for scientific QA systems that produce verifiable answers rather than unsupported summaries.

AutoQABench: A Three-Level UX Benchmark for Automated Evaluation of Open-Ended LLM Responses

Evaluating open-ended large language model responses remains difficult because response quality depends not only on factual correctness and task completion, but also on subjective and scenario-dependent user experience factors. Existing benchmarks and automatic evaluators are effective for coarse-grained assessment, yet often provide limited insight into how different types of quality failures affect user-perceived response quality. To address this gap, we propose AutoQABench, an initial three-level user experience benchmark for automated evaluation of open-ended LLM responses. The benchmark decomposes response quality into three progressively organized levels: basic acceptability constraints, scenario-specific task effectiveness, and preference-sensitive experiential quality. Based on this design, we construct a dataset covering representative scenarios, including summarization, elaboration, emotional support, and responses to misleading premises, together with expert-defined evaluation standards. We also develop an LLM-based modular evaluator that performs stepwise assessment and generates both intermediate judgments and a final rating. Experimental results show that AutoQABench achieves good agreement with expert judgments and supports analysis at the module, scenario, and final-grade levels. Rather than replacing human evaluation, AutoQABench is intended as a scalable auxiliary evaluator that helps identify overlooked risks, task-completion failures, and quality differences in open-ended LLM interactions.