Diversity and Carelessness in LLM-Generated Psychometric Data: Implications for Data Quality and Trust

Abstract

Large language models (LLMs) are increasingly used to generate synthetic, human-like data for research and applied decision-making, including synthetic responses to self-report instruments such as surveys and questionnaires. As these synthetic datasets are folded into pipelines that inform scientific inference and downstream decision support, their trustworthiness becomes a practical concern. In human data, careless responding is a well-documented source of bias in self-report measures, yet no research has examined whether LLMs reproduce similar patterns of carelessness when generating synthetic responses. This study addresses that gap by comparing a large synthetic dataset generated by GPT-3.5-Turbo on a mental health questionnaire to a large human dataset, examining response diversity and multiple indicators of careless responding. GPT-generated scale responses were less variable than human responses, showing limited use of extreme response options, higher repetition, and reduced multivariate dispersion relative to a human-derived reference distribution. In contrast, the textual explanations accompanying the synthetic responses exhibited greater lexical variation, indicating that GPT’s linguistic ability did not translate into an ability to reproduce the heterogeneity of human psychometric response behavior. Carelessness-like patterns were also observed in the synthetic data, although conventional indices behaved differently across datasets and could not always be interpreted as direct evidence of human-like careless responding. These findings raise questions about the reliability of LLM-generated data for research and decision-making applications and point to the need for quality-assurance checks before trusting such data in downstream intelligent systems.