What students ask matters: LLM interaction depth, task quality, and immediate recall in higher education
TL;DR
The findings indicate a dissociation between performance quality and short-term recall in LLM-supported study, which aligns with cognitive-psychology evidence that elaboration improves comprehension while retrieval practice consolidates retention.
Abstract
Large Language Models (LLMs) are rapidly transforming higher education, yet evidence on how interaction patterns affect learning remains limited. This study examines whether explanation-seeking dialogue with an LLM is associated with task quality and immediate recall in a controlled learning session. Twenty-two postgraduate students completed a pre-test, an LLM-assisted neuroeconomics case task, and an immediate post-test. Fine-grained interaction logs captured per-turn telemetry, enabling extraction of Depth (proportion of "why/how/explain" prompts), Volume, and Pacing features. Results showed large immediate pre–post score gains (Cohen’s dz = 2.12, p < .001). In the task-quality regression model, Depth was positively associated with task quality beyond baseline and Volume (β = 6.27, p = .006), indicating that a one-standard-deviation increase in explanation-seeking prompts was associated with approximately six additional marks on a 0–100 scale. Depth was not associated with immediate recall (β = − 0.014, p = .728); instead, gain scores were strongly associated with baseline knowledge, consistent with reduced headroom for improvement among students starting from a higher pre-test score (β = − 0.161, p < .001). The findings indicate a dissociation between performance quality and short-term recall in LLM-supported study. This aligns with cognitive-psychology evidence that elaboration improves comprehension while retrieval practice consolidates retention. Pedagogically, the dissociation suggests that depth-oriented dialogue may need to be paired with deliberate memory-strengthening activities such as self-testing or spaced retrieval if comprehension gains are to translate into recall, although the present design does not test such activities directly. Methodologically, it contributes a replicable, privacy-preserving instrumentation pipeline linking conversational telemetry to learning outcomes. Limitations include single-group design, small sample (n = 22), immediate testing only, and keyword-based depth proxies. Future work should randomise assistance styles, incorporate delayed retention tests, and refine depth measurement through semantic coding.
