Skip to content
Review

Evaluating LLM-Generated Feedback on Handwritten Mathematics: Feedback Quality and Subsequent Exam Performance

Oct 2026 · 0 citations · 25 references
Computer Science

TL;DR

The findings show the potential of an integrated framework merging instructor guidance with automated feedback generation to provide personalised learning support at scale, as well as the potential of an integrated framework merging instructor guidance with automated feedback generation to provide personalised learning support at scale.

Abstract

We evaluate the quality of feedback and practice exercises generated by GPT-5.1 from handwritten university mathematics answers and their grades, and examine whether receiving this feedback improves subsequent exam performance. We use six weekly bonus tests in an English (n=343) and a Dutch (n=236) cohort. Tests 1 and 3 used a crossover feedback design; otherwise, all attendees received feedback. To evaluate the feedback, six reviewers assessed 157 messages from three English-cohort tasks using six criteria scored from 0 to 3. The practice exercises received the highest proportion of strong ratings (95.5%), while diagnostic specificity (41.1%) and clarity and tone (33.4%) received fewer. Feedback for the lowest-scoring answers was rated as less actionable, suggesting that these students may need more targeted guidance from instructors. Comparisons of related exam questions showed mixed results, with no statistically significant differences after adjustment across four comparisons. Greater attendance and feedback exposure were associated with higher exam scores, but the contribution of feedback could not be separated from attendance and practice. Our findings show the potential of an integrated framework merging instructor guidance with automated feedback generation to provide personalised learning support at scale.

View source

Similar papers

Preprint Oct 2026

Beyond Score Accuracy: Examining the Diagnostic Quality of LLM-Generated Structured Assessment in Higher Education

Using the JorGPT dataset of 3,041 student responses to 50 open-ended computer science questions, scored by both human instructors and three commercial LLMs, this study identifies three systematic discrepancies between the apparent and actual quality of LLM-generated grading and feedback.

Xi Zhao, Xin-Yue Jiao, Zhen Xu · 0 citations
Sep 2026

Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence

This study aimed to automate hierarchical diagnostic reasoning (HDR), a constructive method for evaluating practical judgment skills, by developing and testing an evaluation process using a large language model. HDR is a descriptive task that measures higher-order cognitive skills by requiring students to identify and...

Satoshi Takahashi, A. Yoshikawa, Megumi Kose et al. · 0 citations
Review Sep 2026

Vision-Language Models for Criterion-Level Grading of Handwritten Examinations in Outcome-Based Education

The findings highlight the need for rubric-specific calibration, repeatable scoring, review of consequential errors, and separate validation of explanations, and separate validation of explanations for criterion-level assessment research and grading tools with teacher oversight.

Asif Hasan Tonmoy, Saad Ahmed, Md. Khalid Syfullah et al. · 0 citations

Evaluating the Pedagogical Quality of LLM-Generated Feedback: A Criterion-Based and Comparative Study

A multi-dimension 14-item rating instrument grounded in formative feedback theory is developed and used to compare feedback from a multi-agent system, a single-agent system, and an instructor and shows that a multidimensional approach identifies differences in feedback quality that a single overall rating could miss.

Harvey Ngoe Kolle, Carrie Demmans Epp, Amna Liaqat et al. · 0 citations
Open access Sep 2026

Impact of automated scoring on interpreting performance and self-regulated learning: evidence from a pedagogical experiment

Confronting the challenges of feedback in traditional interpreting training, this study explores how an automated scoring system affects student interpreters’ performance and self-regulated learning (SRL). Guided by Zimmerman’s cyclical model, the study employed a 14-week quasi-experimental design with two intact class...

Chen Chen, Ting Liu · 0 citations
Open access Nov 2026

Time Spent on the Feedback Screen and Whether Students Correct Their Errors in a Mathematical E-Assessment Platform

Feedback is central to formative e-assessment. How long students spend on it may itself be associated with whether they correct an error on a second attempt. This study examined the association between feedback-screen time (FST) and second-attempt accuracy (SAA) in mathematics, and whether it varies across items. We an...

Miguel Alcaraz-Rivera, Cristina Eccius-Wellmann · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.