Adaptive Chatbot Significantly Surpasses ChatGPT at Teaching and Assessing User Knowledge about Mental Health and Problematic Social Media Usage
Personal assistants, chatbots, and large language models (LLMs) increasingly pervade our society. While their main function for people is to answer questions by providing information, a new area of focus and issues was identified in this experiment. There were two main issues identified: namely, that chatbots do not adjust their responses to the user and their background, and that user comprehension is not assessed and responses therefore do not adjust to that comprehension. In this experiment, a self-assessment chatbot was created to attempt to solve this issue. 9 middle school and high school students in the United States were randomly assigned to a control group and an experimental group. Each group was given an instructional document containing information about teen mental health and social media usage. The control group was given Chat GPT to study with and the experimental group was given our self-assessment chatbot. Finally, each group was given the same posttest, scored out of 15 points. This posttest contained 15 questions about the subject material. Results showed statistically significant evidence that students who used the self-assessment chatbot scored, on average, 20.7 percentage points higher than those who used Chat GPT to study, equivalent to about 2 letter grades higher.