Exploring LLM-generated hints for students’ Error Recovery in programming

Abstract

This paper explores how students in Higher Education use LLMs for programming and how learning environments should scaffold such support. We conducted a repeated-measures exploratory field study in two programming courses, a Bachelor’s and a Master’s level course, using a JupyterLab environment with LLM support. Students encountered different support conditions: no support, generic error-type support, and tailored support using traceback and source-code context. We studied the impact of these conditions on error recovery. Tailored support was associated with higher error recovery than both no support and generic support, a benefit that held across course levels rather than varying with expertise. These findings may indicate that LLM-supported programming environments should supply context-sensitive scaffolding, automatically providing the model with the learner’s code and error trace rather than relying on the learner to communicate that context.