The Educational Effects of Generative AI in University Programming Courses: Focusing on AI Literacy Enhancement and Changes in the Perceived AI Literacy Gap

Abstract

This study aims to empirically examine the effects of utilizing generative AI (Gemini) in a machine learning programming course for university students majoring in AI-related fields on learners' AI literacy and on the initial gap in perceived AI literacy among the majors. A nonequivalent control group pretest-posttest design was applied to 48 AI-related majors at D University, comprising an experimental group (n=26) and a control group (n=22). Both groups followed the same 15-week machine learning programming curriculum; the experimental group engaged in generative AI-assisted pair programming, while the control group conducted traditional self-directed practice. The results are as follows. First, the pretest revealed that the AI literacy of the experimental group (M=2.80) was significantly lower than that of the control group (M=3.32), confirming a clear initial gap in perceived AI literacy even among the majors. Second, the experimental group demonstrated statistically significant pre-to-post improvements in all sub-factors except computational thinking, as well as in the overall score, with particularly prominent gains in AI-based problem-solving (d_z=0.88) and the social impact of AI (d_z=0.84); however, the control group also showed significant improvements in most areas, so these within-group gains cannot serve as evidence of a treatment effect. Third, an analysis of covariance (ANCOVA) controlling for pretest scores revealed no statistically significant group effect on overall posttest scores (F=2.62, p=.113), whereas the control group remained significantly higher in computational thinking even after controlling for pretest scores (F=5.16, p=.028). In short, no positive treatment effect of generative AI use was identified in the between-group comparison that controlled for pretest scores. This study therefore does not claim an effect of generative AI; it reports only the descriptive finding that the perceived-literacy gap did not widen further in the experimental group, whose initial level was lower. Because the absence of a significant difference is not proof of equivalence, the results should be treated as preliminary. Based on these findings, this study proposes specific instructional design guidelines for major education, including process (prompt)-centered evaluation, the mandatory inclusion of a critical acceptance stage, and the integration of flipped learning.