From Black-Box to Glass-Box: The Impact of Procedural Transparency and Explainability on AI Literacy in K-12 Education

Abstract

As artificial intelligence (AI) becomes ubiquitous, moving learners beyond foundational awareness toward higher-order AI literacy is increasingly essential. Automated AI tools prioritize high-level abstraction, risking “illusory competence” in which operational success masks conceptual misunderstanding. This study evaluates X-Mind, a glass-box tool that integrates procedural transparency and explainability, against the comparatively abstracted environment of Google Teachable Machine, using a quasi-experimental mixed-methods design with 501 Malaysian students (primary, n = 266; secondary, n = 235). The glass-box condition produced significantly greater knowledge gains at both levels. Their locus, however, was moderated by developmental stage. Secondary students achieved significant gains, demonstrating a large advantage in applied ML reasoning and improvements in bias reasoning. Positive gains in representativeness reasoning were also observed after accounting for baseline differences. Primary students achieved significant gains in foundational ML knowledge. The visible workflow provided a transparent learning experience that strengthened conceptual understanding and supported more consistent learning outcomes than fully automated environments. Qualitative feedback indicated that engagement, sustained by the collaborative build activity, accompanied these gains, while critiques centered on information density and language accessibility. The findings reveal a transparency-complexity tradeoff, whereby transparency supports foundational learning in younger learners and expands opportunities for higher-order reasoning among older learners. This emphasizes the critical role of AI interface design in cultivating evaluative agency, suggesting that visibility should be calibrated to developmental readiness rather than uniformly maximized.