AI-SOLO: A hybrid framework for cognitive assessment in programming education

Abstract

Traditional programming assessments focus on correctness and efficiency and do not consider students' cognitive learning processes or code structure. Teachers then encounter challenges in providing effective feedback to facilitate cognitive learning in programming education. This study proposes a hybrid AI framework, AI-SOLO, that combines CodeBERT semantic embeddings, Graph Neural Networks (GNNs), and behavioral metadata to automatically categories student programming submissions. Each submission was mapped to the relevant level of understanding described in the Structure of Observed Learning Outcomes (SOLO) Taxonomy. The framework is multi-dimensional, combining semantic, structural and behavioral evidence of understanding, rather than just output correctness. It was tested in a controlled study with 120 undergraduate students who submitted 1,824 Python programs. AI-SOLO achieved an overall classification accuracy of 91.4% and 83.4% at the Extended Abstract level, demonstrating its ability to classify programming submissions across cognitive levels. The framework also enables immediate, personalized feedback to learners, targeted instructional interventions, and interpretable dashboards for educators to monitor cognitive progress at the class level. The findings indicate that integrating semantic, structural, and behavioral features can support the cognitive-level classification of student programming submissions and provide a basis for structured assessment in programming education.