Testing the Utility of Using Large Language Models to Create Personalized Networks From Therapy Session Transcripts: A Proof of Concept Study

Abstract

Recent advances in psychotherapy have focused on treatment personalization, such as by selecting treatment modules based on individual networks. However, estimating personalized networks typically requires intensive longitudinal data, which is not always feasible to collect. A solution to increase scalability of network-driven treatment personalization is leveraging large language models (LLMs). In this study, we developed an end-to-end pipeline for automatically generating client networks to support case conceptualization and treatment planning. We annotated 8,028 utterances from 77 therapy transcripts (N = 6). In the first stage of the pipeline, we identified clinically relevant processes (binary classification) and their corresponding dimensions (multi-label classification). Then, we introduced a two-step method that grouped the processes into clinically meaningful clusters and generated labels for the clusters. Finally, we generated connections between clusters. Evaluation results generally supported model utility, however, interrater agreement on model performance metrics was inconsistent, ranging from poor to substantial. Qualitative examination of the networks indicated consistency with original study data. Given coherence and interpretability of the generated networks, developing networks from therapy transcripts using LLMs appears to be feasible. Nonetheless, more research is needed to examine whether these networks improve treatment outcomes, including relative to other methods of treatment personalization, such as statistically estimated networks. Potential use cases and limitations of our pipeline are discussed.