Aug 2026· Proceedings of the Canadian Engineering Education Association (CEEA)· 0 citations
TL;DR
A comparative analysis of six LLMs for generating formative feedback on introductory Java programs containing predefined defects under controlled conditions reveals substantial cross-model variation, particularly in multi-defect scenarios.
Abstract
Delivering high-quality formative feedback in large introductory programming courses remains challenging despite the widespread use of automated grading systems. Although large language models (LLMs) can generate natural-language explanations grounded in student code and test outcomes, their diagnostic reliability and pedagogical suitability require systematic evaluation. This study presents a comparative analysis of six LLMs for generating formative feedback on introductory Java programs containing predefined defects under controlled conditions. Model outputs were evaluated across key dimensions of feedback quality and performance. Results reveal substantial cross-model variation, particularly in multi-defect scenarios. More precise fault localization was associated with higher diagnostic accuracy and completeness, but also with more frequent instructional-constraint violations. No single model consistently outperformed others across all criteria, and response latency did not reliably correspond to diagnostic quality. These findings underscore the context-dependent nature of LLM-generated formative programming feedback.
Large language models (LLMs) are increasingly used to evaluate open-ended educational responses. However, their performance is often assessed using aggregate metrics that provide limited insight into prediction stability, uncertainty, error patterns, and feedback quality. This study presents EduFairBench, a reproducible evaluation protocol designed to characterize LLM behavior across short-answer assessment and automated essay scoring using open educational benchmarks. The protocol combines repeated inference, majority-vote consolidation, uncertainty estimation, error analysis, and structural evaluation of generated feedback within a unified experimental framework. Experiments were conducted on SciEntsBank, Beetle, and ASAP2, comprising 2,000 student responses and 10,000 independent LLM inferences. The results showed moderate predictive agreement with human assessment while revealing substantial differences between nominal and ordinal evaluation tasks. Repeated inference demonstrated high internal stability across benchmarks, although systematic errors remained in semantically adjacent categories, indicating that prediction consistency does not necessarily imply correctness. Feedback quality varied by task type, with longer textual contexts yielding more specific and pedagogically structured explanations. These findings demonstrate that evaluating educational LLMs requires complementary analyses beyond conventional performance metrics. EduFairBench provides a reproducible methodology for jointly analyzing predictive performance, robustness, uncertainty, and feedback quality, providing a comprehensive methodological framework for the rigorous evaluation of LLM-based educational assessment systems.
W. Villegas-Ch., Aracely Mera-Navarrete, Fernando Zúñiga-Tello et al.· Frontiers in Artificial Inte...· 0 citations
This paper introduces CodeOwl, an AI-driven tool that automates the generation of tiered Parsons problems automatically, and evaluated CodeOwl with a mixed-method framework comprising complexity analysis, expert ratings, and user studies.
Luca Cisternino, Florian Obermuller, Gordon Fraser· arXiv.org· 0 citations
Large language models are increasingly deployed in education as tutors, teaching assistants, and content generators. These roles place demands that ordinary question answering does not: a usable education-facing model is supposed to be accurate, safe under sensitive prompts, instructionally useful, and aligned with pedagogical goals at the same time. Existing benchmarks evaluate these requirements largely in isolation, so none assesses education-facing suitability as an integrated profile. We introduce ELBench, the first benchmark to evaluate all four requirements (General Capability, Safety and Trustworthiness, Basic Education, and High-Level Cultivation) on the same models under a common protocol, combining curated public sources with newly synthesized safety and cultivation data. We evaluate nine models, seven frontier general-purpose systems and two education-specialized variants, and report three findings. First, module-level profiles are more informative than a single aggregate: the top six models are statistically indistinguishable on overall score, yet their module leaders differ substantially, and safety is anti-correlated with practical teaching (r = -0.83). Second, the Chinese-developed models lead the safety module, the most discriminative in the suite; this advantage is largest on region-specific normative content and narrows, but does not vanish, on universal-harm content. Third, the two education-specialized models lead neither education module, and on High-Level Cultivation all models share a systematic blind spot: on the structured judgment task they converge on the same non-reference option, favoring pedagogical style over fit to the stated goal, so the module scores uniformly low and does not separate models. This raises, but does not resolve, whether domain post-training keeps pace with frontier systems on education tasks.
Yi-Lin Jiang, Xiao-Rong Zhu, Fei Tan et al.· 0 citations
Test-based learning is effective in fostering knowledge retention, but manually creating assessment questions remains time-consuming and limits personalized student practice. The advent of Large Language Models (LLMs) has introduced new possibilities for Automatic Question Generation (AQG). Motivated by this context, this survey provides a comprehensive overview of AQG using LLMs, focusing on educational applications. Following the PRISMA methodology, we reviewed 132 studies published between 2023 and 2025. Our contributions include a taxonomy of question types by response openness, an analysis of AQG efforts across knowledge fields, educational levels, evaluation strategies, and difficulty control. We also identify recurring challenges and research opportunities.
Roberto Oliveira, A. Hernández, M. Garbin et al.· ACM Computing Surveys· 0 citations
Traditional programming assignment assessments primarily focus on evaluating the correctness of the final output, often neglecting critical aspects such as engagement, collaboration, and debugging skills. As programming education increasingly shifts toward project-based and collaborative learning, there is a growing need for more holistic evaluation methods. This paper proposes a hybrid conceptual framework that integrates cognitive correctness metrics with behavioral analytics derived from Source Code Management (SCM) systems, such as GitHub. The framework captures authentic student behaviors during assignment development and leverages machine-learning models combined with explainable AI techniques to transparently predict grades. This conceptual study advocates a paradigm shift from purely product-based assessments to a more process-oriented, learning-centered evaluation model that promotes deeper and more meaningful educational experiences in programming education.
Unknown authors· International journal of com...· 0 citations
The time complexity of algorithms is a critical concept in computer science and engineering, and it is recognized as a fundamental topic in the Association for Computing Machinery (ACM) curricular recommendations. In alignment with these guidelines, numerous universities worldwide incorporate this topic into their introductory computer science courses, which typically have large student enrollments annually. Consequently, there is an increasing demand for automation in both instructional and assessment processes. This study explores the potential of Large Language Models (LLMs) to assist teaching staff in generating source code segments with predefined time complexity and determining the time complexity of given code segments, with applications in educational and examination contexts. We proposed a novel methodology for LLM evaluation in the aforementioned context and evaluated three prominent LLMs: ChatGPT, Gemini, and Llama, on their ability to generate and analyze C code segments exhibiting linear, logarithmic, quadratic, and exponential time complexities. A framework was developed to automate the prompt and segment generation and time complexity determination using two mainstream prompt engineering methods: zero-shot and chain-of-thought, and assessed the differences in code generation and time complexity analysis. A total of 960 generated segments were assessed on the correctness of time complexity, structural appropriateness, and suitability for exam use. The results suggest that ChatGPT is the most suitable LLM for generating segments with predefined time complexity (success rate goes up to 61%). All LLMs yielded the best results in generating linear segments, while exponential complexity posed the greatest challenge overall. A subset of generated segments was extracted to evaluate the time complexity determination capabilities. All three LLMs were asked to find the time complexity of each extracted segment. The most accurate LLM is ChatGPT (79.6%). We also assessed how good each LLM is in determining the time complexity of segments generated by itself. Llama outperforms others in that task (83% of successful determinations) when the zero-shot prompt method is used. The findings suggest that current LLMs cannot fully automate question generation and time complexity problem solving. However, they can substantially support the process and reduce the workload for educators.
Đorđe Pešić, Milena Vujošević Janičić, Marko Mišić et al.· PeerJ Computer Science· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
