Analysis of Prediction Accuracy of Different Chatbots for Big Five Personality Traits in High School Students

Abstract

Chatbots based on large language models (LLMs) are commonly used by teenagers for daily communication and introspection. All the previous studies merely compare personality outputs created by a certain AI with no numeric assessment of differences from the self-reported personality scores of students. Questionnaire data are obtained from 33 Chinese students who study in Canada-based education programs. BFI-10 ten-question short inventory is selected for measuring self-reported personality trait scores of the participants which are compared to personality prediction scores by five major chatbots. Participants provide subjective evaluation regarding the ability of chatbots to understand their personality traits. Data cleaning, calculating errors and visualization of the results are conducted with Pandas, Numpy and Matplotlib in Python. Two research hypotheses are proposed before conducting data analysis. It is found that the AI-based personality trait scores are still relatively correlated with the results of self-assessment of participants. There are some discrepancies in conscientiousness and openness but the highest differences occur in case of neuroticism. Moderate negative correlation is found between subjective evaluation of AI understanding and prediction error. This research provides practical data for further studies of LLM personality prediction in adolescents and proposes certain directions for further investigation.