Findings indicate that behavioral signals present at the very start of an exercise contain useful clues about whether a student will ultimately solve the problem, and that keystroke-level editing logs provide additional value for early prioritization beyond execution logs alone.
Abstract
This paper investigates the feasibility of early detection of struggling students during CS1 programming exercises using keystroke-level logs. Some students fail to reach a correct solution before the exercise ends, and by the time this becomes apparent from grades or final outcomes, the opportunity for timely instructor support aimed at helping them recover may have passed. We use data from the CodeBench platform, which records real-time code editing events at the keystroke level, alongside execution and submission logs. We define two outcome groups: Breakthrough (BT) students, whose prior submissions all receive 0% and whose final submission achieves full credit, and Fully Stuck (FS) students, whose submissions all receive 0% without reaching a correct solution. To examine this feasibility, we focus on two questions: (RQ1) whether adding keystroke-level editing features improves the prediction of FS students over execution-log features alone, and (RQ2) at which stage BT and FS students can be predicted most accurately. Experiments on the 2019-1 semester of the CodeBench dataset, comprising 507 students, compare three feature configurations: execution-based features (ExecOnly), CodeMirror-based features (CMOnly), and their combination (Combined). We evaluate prediction across successive submission-based stages during each exercise. In the earliest stage, CMOnly outperforms ExecOnly (AUROC 0.654 vs. 0.575), and Combined further improves over ExecOnly by +0.098 (AUROC 0.674). Across all configurations, the earliest stage yielded the strongest predictive signal. These findings indicate that behavioral signals present at the very start of an exercise contain useful clues about whether a student will ultimately solve the problem, and that keystroke-level editing logs provide additional value for early prioritization beyond execution logs alone.
The difference between a student who thrives in computer science and one who grows to dislike it often comes down to their debugging experiences. Until recently, debugging was not an explicit focus of computer science pedagogies; instead, it was assumed students would internalize debugging skills through the practice of programming. Yet, learning to debug can be challenging for novice students and may demotivate them if they are not carefully supported throughout the process. Debugging requires students to engage with multiple skills, such as program comprehension, code reading, writing, tracing, predicting, and modifying. While debugging pedagogies are becoming more common, they remain understudied in elementary education, particularly in block-based environments like Scratch. In this qualitative case study, we explored how 4th and 5th grade students described identifying bugs after participating in a four-week online Scratch Summer camp that included explicit debugging instruction. One week after the camp, six students completed six structured Scratch debugging tasks and participated in retrospective think-aloud interviews. Thematic analysis revealed four bug identification strategies: (1) identifying missing or incorrect blocks, (2) theorizing what caused the bug, (3) visualizing bugs through animation, and (4) clarifying the intended program behavior. These strategies suggest that while many learners can identify block-level bugs, fewer describe intended behavior or explain why the program behaves as it does. We discuss implications for designing K–-5 debugging instruction that supports students in moving from identifying bugs in the code to reasoning about program behavior.
Yerika Jimenez, Christina Gardner-Mccune, K. Tong et al.· Annual Conference on Innovat...· 0 citations
This paper introduces CodeOwl, an AI-driven tool that automates the generation of tiered Parsons problems automatically, and evaluated CodeOwl with a mixed-method framework comprising complexity analysis, expert ratings, and user studies.
Luca Cisternino, Florian Obermuller, Gordon Fraser· arXiv.org· 0 citations
This paper explores how students in Higher Education use LLMs for programming and how learning environments should scaffold such support. We conducted a repeated-measures exploratory field study in two programming courses, a Bachelor’s and a Master’s level course, using a JupyterLab environment with LLM support. Students encountered different support conditions: no support, generic error-type support, and tailored support using traceback and source-code context. We studied the impact of these conditions on error recovery. Tailored support was associated with higher error recovery than both no support and generic support, a benefit that held across course levels rather than varying with expertise. These findings may indicate that LLM-supported programming environments should supply context-sensitive scaffolding, automatically providing the model with the learner’s code and error trace rather than relying on the learner to communicate that context.
Bibeg Limbu, Kaimao Sheng, Mona Münstermann et al.· Proceedings of Mensch und Co...· 0 citations
Novice programmers often have flawed mental models of program execution, and knowing what misconceptions they are likely to hold can help instructors to prevent and correct these misconceptions. Recently, tools have emerged which task students with simulating a program's execution and constructing the run-time memory layout. Such tools create new opportunities to systematically log student errors and identify flaws in their mental models --- including low-level misconceptions that may not be visible when looking at other artifacts such as student-written programs. This paper analyzes log data that was collected by instrumenting one such tool. We identify the most common mistakes that students made, as well as mistakes that were particularly likely to persist even in the face of targeted feedback. We also present the results of a conceptual pretest-posttest designed to help contextualize these mistakes.
Michelle Le Pham, Rebecca Schreib· Annual Conference on Innovat...· 0 citations
Background and Context. Large Language Models (LLMs) have become widely accessible to students in introductory programming courses [1], yet limited research evaluates their performance on authentic assignments with pedagogical constraints such as restricted language features and course-specific conventions. Existing benchmarks target capabilities that do not reflect real coursework: function synthesis [2], competition programming [4, 7], repository-level bug fixes [5], and class-level Python generation [3]. Novice programmers tend to provide minimal context and rely on single-shot interactions [1, 6], motivating our empirical experimentation with holistic prompting alongside a compositional strategy. Objectives. This study investigates (1) how current LLMs perform on introductory C++ programming assignments in terms of functional correctness and response behavior, and (2) the extent to which code generated by different models exhibits structural similarity, with implications for academic integrity detection. Method. Two multi-class C++ programming assignments were selected from a CS2 course covering dynamic memory management, class design, and complex data structures involving tree traversal and reference-counted shared ownership. We generated 62 solutions to the two assignments using models in the Claude, GPT, Gemini, and DeepSeek families. The same autograder suite used to grade student code was applied to LLM-generated solutions. For Claude and GPT, results were further compared against 1,436 student submissions from a prior course offering. We also measure the probability that its generated solution fully passes all functional tests on the first attempt using the Pass@1 metric. Finally, the winnowing algorithm is applied to evaluate structural similarity across LLM-generated solutions. Findings. On the first submission compiling with only accepted libraries, autograder scores ranged from below 60% to 94%, bimodal on the complex assignment and normal on the simpler one, while Pass@1 ranged from 0–7% and 33–100%, respectively. Enabling “thinking” mode yielded no improvement for GPT and hurt Claude Sonnet 4.5, which scored 17% lower on average. LLMs reliably handled memory management and constructors, but struggled with the test cases students found hardest and consistently failed C++20 style standards. Models differed in response behavior: Claude followed compositional instructions precisely; GPT frequently implemented more than requested; Gemini embedded reasoning and citations into code comments; DeepSeek produced verbose self-dialogue with multiple code versions per response. Within-family structural similarity substantially exceeded cross-family similarity, and all models converged on near-identical solutions upon reaching a perfect score. Implications. LLM performance on CS2 assignments depends more on task complexity and prompt strategy than on model choice, suggesting that educators should design assignments with constraints that resist one-shot automation and encourage iterative, scaffolded engagement. The high within-family structural convergence of LLM-generated code challenges the sufficiency of existing similarity-based academic integrity tools, necessitating new detection strategies that account for characteristic patterns across model families.
Iris Xu, Michał Nowak, E. Shaffer· Proceedings of the 2026 ACM...· 0 citations
While LLM-rewritten messages significantly improved subjective evaluations, with pragmatic messages rated as clearer and less cognitively demanding, these perceived gains did not translate into statistically significant improvements in objective debugging performance.
Alexandru-Radu Moraru, Shreyan Biswas, U. Gadiraju· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
