Chatbot Engagement Does Not Always Beget Metalearning: Evidence from Three Countries
TL;DR
A preregistered, three-country randomized experiment on out-of-context image misinformation, manipulating a correction's channel affordances (synchronicity, bandwidth) across four conditions shows engagement mechanisms do not substitute for slow AI literacy.
Abstract
Chatbots deliver real-time fact-checks, but whether a chatbot correction leaves anything behind once the chatbot is gone - metalearning, distinct from correcting misbeliefs - is untested. We report a preregistered, three-country randomized experiment (USA, India, Singapore; N ~ 2,200) on out-of-context image misinformation, manipulating a correction's channel affordances (synchronicity, bandwidth) across four conditions: Control, Links-only, Static explanation, and a Socratic Chatbot built on a validated out-of-context detector, with an unaided retest one week later. The Chatbot produced the largest immediate discernment gain (d = 0.097, p = .023). All three interventions reduced sharing of false claims (d ~ -0.12, p<.01). One week later, no advantage persisted: the Chatbot arm declined relative to Control, most sharply in India and Singapore, and in India on claims it never discussed. Decay tracked affordance level and did not vary by country. Engagement mechanisms, we argue, do not substitute for slow AI literacy.
