A Problem-Level Cognitive Framework for Analyzing Learning Performance: An Empirical Application in AI-Supported and Teacher-Mediated Statistics Instruction

Abstract

Students with similar overall test scores may nevertheless succeed or struggle with very different kinds of problems. Aggregate performance measures can conceal these differences when assessment problems require different forms of reasoning or different levels of structural complexity. Existing cognitive theories and educational taxonomies describe important aspects of knowledge, reasoning, and complexity, but were not designed to classify individual problems simultaneously according to these two characteristics. This study therefore develops a purpose-built Problem-Level Cognitive Framework that represents assessment problems along two analytically independent dimensions: the dominant reasoning required for a conceptually adequate solution—Formal, Procedural, Theoretical, or Critical—and the structural complexity of the minimal valid solution path, expressed through seven ordinal difficulty levels (D1–D7). The framework was empirically examined using semester-long data from 121 undergraduates enrolled in one of two instructional conditions: AI-supported or teacher-mediated introductory statistics instruction. Binary outcomes for each student–problem pair were analyzed using generalized linear mixed-effects models with crossed random effects for students and problems. Successful problem solving varied substantially across reasoning dimensions, difficulty levels, and assessment timings. In contrast, instructional condition made little explanatory contribution, and neither the condition-by-reasoning nor the condition-by-difficulty interaction was significant. The framework thus revealed a stable cognitive organization of performance across two substantially different instructional environments, while also identifying pronounced variation among cognitively differentiated problems. By separating the form of reasoning governing a valid solution from the structural complexity of the required solution process, the framework transforms heterogeneous assessment problems into cognitively comparable analytical units. It provides an operational basis for investigating problem-level performance structures and for examining their stability across learners, assessments, instructional environments, and educational domains.