A context-specific workflow was developed that translated framework analysis into bounded, sequential LLM-supported tasks and applied it to 20 Chinese-language interviews on fear of dementia, where Qwen-output showed the most consistently favourable overall profile, while Claude-output was most often ranked first in individual domain-dimension evaluations.
Abstract
Large language models (LLMs) are increasingly used for qualitative analysis, but strong language performance does not guarantee strong interpretation when meaning depends on method and context. We developed a context-specific workflow that translated framework analysis into bounded, sequential LLM-supported tasks and applied it to 20 Chinese-language interviews on fear of dementia (approximately 150,000 characters). Four multilingual LLM-generated outputs and a previous researcher-only analysis were blindly evaluated by 16 relevant experts across different analytical domains and qualitative quality dimensions (496 expert-item records). All LLMs completed the workflow, but rankings varied markedly. Qwen-output showed the most consistently favourable overall profile, while Claude-output was most often ranked first in individual domain-dimension evaluations. The researcher analysis received both highly favourable and highly unfavourable rankings; no output was consistently preferred. LLM-assisted qualitative analysis should be evaluated within its research contexts, with transparent workflows, multidimensional assessment, and continued researcher oversight.
This article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs by describing underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question.
S. E. McKinney, Phuc Vu, Samuel A. Justice et al.· arXiv.org· 0 citations
The hypMOOVE-PHC study is the hypothetical vignette phase of the Massive Open Online Validation and Evaluation (MOOVE) initiative, implemented in Kenya, Malawi, and Tanzania, and aims to validate a pool of LLMs through clinical review of expert-generated vignettes through fully crossed repeated-measures comparative eva...
P. Macharia, C. Kachimanga, M. Mahende et al.· medRxiv· 0 citations
Objective
This methodological validation study evaluated a large language model (LLM), multi-step framework for thematic analysis (TA) of healthcare interviews, compared to a human-only coded reference standard.
Materials and Methods
The unit of analysis was deidentified transcripts of interviews with geriatrics pati...
J. Butler, Makoto Jones, C. Balbin et al.· JAMIA Open· 0 citations
Current evidence supports cautious deployment of LLMs in selected healthcare tasks under structured oversight, and the proposed five-dimensional framework coupled with a three-tier risk model is intended to support researchers and healthcare organisations in assessing readiness and implementing LLM-enabled tools respon...
J. C. Ferreira, Isabel Rosa· Frontiers in Digital Health· 1 citation
D-study simulations support the use of hybrid human-LLM workflows to reach coding consensus for deductive reasoning tasks through mode responses, an alternative consensus-generating mechanism to traditional adjudication discussions.
Emily Rush, M. Karim, George S. Yacu et al.· Medical Teacher· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.