What Users Say About Reimbursable Digital Therapeutics in Germany: Large-Scale App Store Review Analysis Using a Large Language Model

Abstract

Abstract Background In 2019, Germany introduced a unique regulatory framework for digital therapeutics (DTx), termed digital health applications (DiGAs) in Germany, with the goal of integrating evidence-based DTx into statutory health care. DTx are eligible for reimbursement by statutory health insurance if manufacturers demonstrate positive health care effects, such as improved health status or health literacy, in a controlled study setting. Although regulatory evaluation primarily relies on manufacturer-conducted studies, these studies do not fully capture how users experience and use DiGAs in everyday life. Objective The study aims to systematically characterize user experiences with reimbursable DiGAs by analyzing the sentiment and thematic content in publicly available app store reviews at scale to identify recurring patterns of positive and negative feedback across regulatory-relevant quality dimensions. Methods All DiGAs with publicly available mobile app reviews were identified via the official German Federal Institute for Drugs and Medical Devices (BfArM) directory. User reviews were extracted from the German Apple App Store and Google Play Store using a tailored Python script. Reviews were then processed using GPT-4o, which extracted up to 5 core statements per review and assigned each a sentiment label (positive, neutral, or negative) and one of ten predefined thematic categories to each statement: content, technology, cost and reimbursement, login and registration, prescription and approval, user experience and design, support, tracking and documentation, effectiveness, and overall impression, all derived from the regulatory and quality criteria applicable to DiGA certification. Classification quality was assessed through manual validation, in which each of the 3 authors independently reviewed one-third of all model outputs. Results In total, 44 mobile DiGAs were included and analyzed. After data extraction and cleansing, the final dataset comprised 4328 reviews containing at least one interpretable statement, resulting in 9439 interpretable statements. A systematic validation of the automated classification demonstrated exceptionally high model performance, with 99% accuracy for sentiment classification ( F 1 -scores of 1.00 for positive and 0.99 for negative categories) and 95% accuracy for category classification (average F 1 -score of 0.95). While the categories overall impression (1233/1431, 86.2% positive) and effectiveness (1440/1609, 89.5% positive) received particularly positive feedback, users commented most negatively on the login and registration process (263/280, 93.9% negative) and technology-related aspects (563/668, 84.3% negative). Conclusions User feedback on reimbursable DiGAs is predominantly positive, particularly regarding perceived effectiveness, overall impression, and content. However, recurring criticism of login and registration processes, technical reliability, and prescription and approval procedures reveals persistent barriers to access and use. As many of these aspects are prerequisites rather than peripheral convenience issues, they should be treated as essential implementation requirements. Analyzed at scale with large language model-based methods, publicly available app store reviews can complement formal evaluation by providing user-centered real-world evidence for postmarket monitoring and regulatory quality improvement.