Integrating patient narratives and clinical indicators for monitoring suicidal ideation in adolescents with major depressive disorder: a human-in-the-loop validation study of LLM-assisted journaling

Abstract

Suicide-related distress in adolescents with major depressive disorder (MDD) can fluctuate between outpatient visits and may not be fully disclosed during brief encounters. Patient-generated diary narratives may capture daily emotional context, whereas structured psychiatric indicators provide standardized clinical information. We evaluated whether integrating these sources within a clinician-supervised, large language model (LLM)-assisted journaling workflow could support between-visit monitoring. We conducted an observational validation study in 108 adolescents and young patients with DSM-5-TR-diagnosed MDD receiving outpatient care in the Republic of Korea. Cross-sectional analyses used the first diary per participant; sensitive-topic evaluation used all 709 valid entries. Structured measures assessed suicidal ideation, depressive symptoms, anxiety, and perceived stress. The self-report outcome was Patient Health Questionnaire (PHQ)-9 item 9 positivity (scores 1–3 versus 0). We evaluated patient-level associations, cross-sectional classification with and without diary narratives, exploratory longitudinal classification across repeated entries, and sensitive-topic detection against clinician annotation. GPT-4o and DeepSeek-V3 were benchmarked against blinded mental health professionals. The integrated MindfulDiary workflow combined pre-journaling screening, in-journaling LLM detection, and clinician alerting. Participants had a mean age of 15.8 years; 58 (53.7%) were female. In patient-level logistic regression, Scale for Suicide Ideation (odds ratio 1.15, 95% CI 1.02–1.28) and depressive symptoms (odds ratio 1.09, 95% CI 1.00-1.19) were associated with PHQ-9 item 9 positivity. Adding diary narratives to depressive symptom scores improved F1-score from 0.79 to 0.81 for GPT-4o and from 0.74 to 0.78 for DeepSeek-V3. In the primary aggregate longitudinal analysis, performance increased with diary count for both LLMs. A sensitivity analysis using the same 16 participants across diary counts showed positive associations for both LLMs and mental health professionals. For clinician-annotated in-journaling sensitive topics, LLM detection achieved an F1-score of 0.85. The integrated workflow captured 193/197 composite-positive entries (98.0%; F1-score 0.96). Patient-generated narratives and structured psychiatric indicators provided complementary signals for monitoring PHQ-9 item 9 positivity and clinician-reviewed sensitive content in adolescent MDD. LLM-assisted journaling may support clinician-supervised monitoring but is not an autonomous suicide-risk classifier. Evaluation across sites and languages, prospective studies, and patient-centered evaluation are needed before routine adoption. Not applicable; the study was not registered.