A multifactor model for assessing programming knowledge based on the synchronization of AI analytics and expert evaluation

Abstract

Introduction The widespread adoption of generative AI tools has created new methodological challenges for the validity of assessment in higher education. If a neural network can write code, how can a student's actual knowledge be assessed objectively? This study proposes a multifactor AI-assisted assessment model for the fair evaluation of AI-supported learning practices in programming courses and examines its alignment with instructor assessment. Methods The study involved 187 undergraduate students from three universities in Kazakhstan. In Phase 1, student submissions were evaluated using a traditional one-dimensional approach focused on functional correctness, and the results produced by instructors and by three large language models (GPT, Claude and Gemini) were compared. In Phase 2, a multifactor assessment model incorporating functional correctness, explanation quality and AI-use level was introduced, and the same submissions were re-evaluated with all three models. Results The baseline analysis showed that AI-assisted assessment assigned systematically higher scores than instructors and demonstrated only moderate agreement with instructor evaluation. Under the multifactor model, agreement with instructor assessment improved significantly for one of the three models (Claude: r = 0.478 →0.647; ρ = 0.449 →0.690; MAE = 18.32 →7.34), while a second model (Gemini) showed a significant reduction in prediction error without a change in rank-order agreement, and a third (GPT) showed no significant change on any measure and reversed the direction of its systematic bias. Even in the best case, agreement remained moderate (approximately 42% of shared variance with instructor scores). Discussion The findings indicate that the effectiveness of AI-assisted assessment depends on the structure of the assessment framework and on the underlying language model, and that a richer rubric does not benefit all models equally. By extending evaluation beyond code correctness toward explanation quality and transparent AI-supported problem solving, the proposed model offers a more balanced and pedagogically grounded approach to assessing undergraduate programming competencies.