The influence of socratic dialogue within GenAI on critical thinking and technology perception

Abstract

The increasing accessibility of Artificial Intelligence (AI), particularly Generative AI (GenAI) and chatbots, highlights the need for users to develop critical thinking skills to evaluate information and understand these systems. While critical thinking is essential for interacting with such technologies, current AI literacy efforts lack robust empirical validation, and effective educational methodologies have yet to be explored. This study explores the potential of the Socratic Method, a structured dialogue-based approach that fosters reflection and critical questioning, to support the development of critical thinking within a GenAI environment. Recognizing that many technology-enhanced and Socratic learning environments often overlook the role of learners’ domain expertise and prior knowledge in shaping effective questioning strategies, the study further compares within-domain and outside-domain Socratic dialogues to evaluate the pedagogical effectiveness of both conditions. Eighty participants, comprising higher education students and teachers, took part in workshops where they interacted with GenAI using two Socratic-style prompts, one within-domain and one outside-domain, and completed questionnaires assessing critical thinking, technical competence, and trust in technology after each interaction. Findings highlight the value of guiding GenAI interactions through Socratic reasoning within domain-relevant contexts to enhance perceived critical thinking, while also stressing the challenges of influencing user trust and perceived competence in technology.