Scaling-up Analysis of Text-Data in Hospitals: An LLM Approach

Abstract

Hospital patient feedback reports offer critical insights for improving care quality and patient satisfaction, yet challenges in interpreting complex narratives and linking insights to outcomes limit their impact. This study utilizes Large Language Models (LLMs) to analyze a dataset of over 7,583 feedback reports from patients based on their emergency department (ED) visit at a large hospital system in southeastern United States. The analysis workflow included de-identification of the reports, followed by regression-based analysis to predict the likelihood-to-recommend score based on metadata, such as arrival-to-room time, patient acuity, and patient demographics. Additionally, an LLM was prompt-tuned to identify the presence (or absence) of specific themes in the free-text comments: trust in staff, expectations being met, feeling of safety, and cleanliness; and if the experience was positive. Results thus far show that arrival-to-room time, ED overcrowding and patient age are significant predictors of the likelihood-to-recommend score. LLM results showed 42.54% indicated trust in staff, 42.32% reported expectations being met, 42.63% noted a feeling of safety, 93.21% observed cleanliness, and 46.23% had a positive experience overall.The next steps in this ongoing study will explore further LLM-generated themes in the reports, and statistical relationships between the metadata and qualitative themes. By combining qualitative narratives with quantitative data, the study provides context for metrics based on the patient’s experience, enhancing the understanding of systemic factors affecting care.  Findings suggest that LLMs can help scale analysis of patient feedback databases toward developing actionable insights to improve care quality.