Adoption and Impacts of a Socratic Generative AI Tutor in Undergraduate Algebra

Abstract

Instructors looking to support personalized mathematics learning may provide access to generative AI tutoring tools. Few studies explore the effectiveness of these systems in classroom settings. This is particularly important in mathematics, where genAI is a shift from historic adaptive intelligent tutoring systems. We evaluated undergraduate algebra students’ (n=58) use of the Socratic generative AI tutor, Khanmigo, through a quasi-experimental mixed-methods study using survey, registrar, grades, and software usage data. One course section was assigned four Khanmigo activities throughout the semester (experimental), while the other was provided access and an introductory activity only (control). A Wilcoxon Rank Sum test showed that experimental students engaged in significantly more chats than control students (p<0.05), controlling for in-class activities. Instrumental help seeking was positively correlated with Khanmigo use, while intentions of seeking help from technology sources were negatively correlated with use. Khanmigo use did significantly predict mathematics grades after controlling for prior GPA and course section. Most students reported using Khanmigo for homework help and exam prep, though lack of visuals and the Socratic design limited perceived usefulness for some students. Integrating genAI tutoring systems into class activities and setting student expectations may improve the likelihood of students using them effectively for learning.