Can We Really Leverage AI-Assisted Automatic Item Generation for Psychological Scale Development?

Abstract

With the emergence of large language models (LLMs), researchers can use chatbots such as ChatGPT to rapidly generate large numbers of items for psychological scale development. This study examines the quality of AI-generated items. Across four preregistered studies, we compared AI-generated items with human-generated benchmark scales in terms of factor structure, reliability, and validity. Items were generated by four LLMs using three prompting strategies for three diverse constructs: global self-esteem, mindfulness, and ``resolution,'' a newly proposed facet of cognitive fluency that reflects people’s desire for immediate and concrete outcomes. The initial item pools contained many low-quality items, including near-duplicates of existing items, semantically redundant items, and, most seriously, items tapping into unintended constructs. After these low-quality items were removed, the resulting AI-generated scales showed strong validity across all conditions, but their factor structure and reliability varied by construct. AI-generated self-esteem scales matched the human benchmark in terms of factor structure and reliability, resolution scales showed intermediate performance, and mindfulness scales performed worst. These findings suggest that LLMs can accelerate item generation but cannot replace expert screening and empirical validation. These limitations may arise partly because general-purpose LLMs are not fine-tuned to distinguish among closely related psychological constructs.