AI-Assisted Analysis of Sentiment Changes Among Repeat Users of Digital Peer Support: A Retrospective Longitudinal Study

Abstract

Background: Digital peer support (DPS) is a scalable tool to provide emotional support across diverse user populations. Previous studies were limited to single-session outcomes, while the impact of repeated DPS sessions is not well understood. Objectives: This study evaluated sentiment trajectories among repeat users of a 24/7, anonymous, real-time moderated DPS service and examined whether DPS engagement across repeated sessions was associated with changes in negative sentiments (depression, loneliness, despair, grief, and work-related stress). User engagement patterns across different conversation topics, and topic-specific differences in DPS use were also explored. Methods: We conducted a retrospective analysis of 13,686 anonymous live DPS chat sessions (January 2023 to December 2024) from 8084 distinct Supportiv DPS users. Of these, 4099 chat sessions from 596 unique repeat users met the predefined inclusion criteria of at least two sessions with a sentiment score ≥5, representing at least moderate expression of the negative sentiment of interest, and a minimum of three scored messages per session. Sessions were categorized into mental health topics using a large language model, GPT-4o-mini. For each topic (depression, grief, loneliness, suicidality, and work-related), relevant sentiment intensity scores were generated by using a few-shot approach based on scale definitions. Individual message-level sentiment trajectories were descriptively characterized for each user using ordinary least squares (OLS) regression across accumulated conversation time. User engagement patterns, session characteristics, and self-reported demographics were compared using nonparametric tests. Results: Over the two-year study period of repeat DPS users, among user-level trajectories with statistically significant OLS slopes, the proportion with negative slopes was 90.9% for despair, 86.1% for depression, 85.7% for grief, 82.4% for work-related stress, and 72.7% for loneliness. Approximately 40–55% of all negative trajectories reached statistical significance. Suicidality-related sessions were ~20% longer and shared ~35% more resources, while loneliness sessions included 14% more resources than other topics. Users focusing on loneliness and suicidality had longer return intervals (11.8 and 18.6 days, respectively) compared to the average (10.4 days). Conclusions: Analyses showed that some users demonstrated downward message-level sentiment across accumulated conversation time. Given its accessibility and scalability, DPS is a promising mental health support tool. Prospective controlled studies are needed to determine whether repeated DPS use produces sustained benefits.