Hierarchical Modeling of AI-Supported Mathematical Problem Solving, Student Diagnostic Reasoning, and Mathematics Achievement

Abstract

This study develops and empirically evaluates a multilevel analytical framework linking AI-generated mathematical reasoning, students’ diagnostic reasoning, and mathematics achievement in AI-supported mathematical problem solving. A classroom intervention with 76 undergraduate Calculus A students generated 380 repeated student–problem observations across five mathematical tasks and three instructional formulations. Students evaluated AI-generated solutions for final-answer correctness, methodological correctness, and procedural completeness, identified mathematical errors, and proposed corrections before completing an individual posttest. Hierarchical mixed-effects models and student-level achievement analyses were used to examine successive stages of the learning process. Instructional formulation was not significantly associated with the quality of AI-generated mathematical reasoning. Final-answer correctness, methodological correctness, and procedural completeness showed substantial task-specific variation. Instructional formulation was significantly associated with students’ error detection, with higher detection performance under Instruction Types B and C than under Type A, while AI-generated response quality remained comparable across instructional formulations. Baseline mathematical proficiency was the strongest predictor of posttest performance. Students’ Mean Error-Detection Event Rate was positively associated with posttest achievement. No robust overall independent association between instructional formulation and posttest achievement was identified after baseline adjustment. Students’ diagnostic engagement with AI-generated mathematical reasoning was associated with subsequent mathematics achievement, supporting the analysis of diagnostic reasoning as a distinct component of AI-supported mathematics learning.