This work introduces PsyEval, a benchmark specifically designed to evaluate LLMs in mental health-related tasks across three core dimensions: knowledge, diagnosis, and emotional support, and reveals considerable gaps in LLMs' current ability to reason accurately and respond appropriately in mental health contexts.
Abstract
Evaluating large language models (LLMs) in the mental health domain presents distinct challenges due to the subtle, context-dependent, and subjective nature of psychological symptoms. We introduce PsyEval, a benchmark specifically designed to evaluate LLMs in mental health-related tasks across three core dimensions: knowledge, diagnosis, and emotional support. PsyEval is constructed to reflect the complexity of mental health scenarios and provides a structured framework for assessing model performance within this sensitive domain. Using PsyEval, we evaluate eleven advanced LLMs with different prompting strategies to investigate how prompting affects their responses. The results reveal considerable gaps in LLMs' current ability to reason accurately and respond appropriately in mental health contexts, while also indicating promising directions for future model enhancement.
Large language models are best understood as emerging assessment-support tools rather than replacements for clinical evaluation because the limited pace of academic validation means that, at present, LLMs are best understood as emerging assessment-support tools rather than replacements for clinical evaluation.
Katie Aafjes-van Doorn, Francine Cheng Ty, A. Hua et al.· Journal of Psychopathology a...· 0 citations
Large Language Models (LLMs) such as ChatGPT and Gemini are increasingly used to answer medical and psychological questions, yet systematic evaluations across domains with differing reasoning demands remain limited. We present a multi-domain expert evaluation of two state-of-the-art models, ChatGPT Pro (v5.2) and Gemin...
Pragna Prahallad, Pranathi Prahallad, Dhrithi P. Desai· 2026 6th International Confe...· 0 citations
Depression and anxiety disorders are among the most prevalent and debilitating mental health conditions worldwide, imposing substantial personal, social, and economic burdens. Although recent advances in Large Language Models (LLMs) have shown promise in supporting mental health assessment and intervention, existing ap...
Jashraj Jani, Sara Akif, Wassila Lalouani· International Conference on...· 0 citations
It is shown that LLMs can assess most major forms of psychopathology from mere minutes of audio, and these results support scoring open-ended narratives with LLMs as a scalable, portable method to translate idiographic diagnostic data into standardized psychiatric assessments.
Whitney R. Ringwald, Aman Taxali, Michael Angstadt et al.· Psychological Medicine· 0 citations
Evaluating 20 frontier and open models under a cross-vendor panel of three LLM judges, it is found that a statistically tied frontier cluster, measurable refusal behavior in two models, and near-identical rankings across judges are found.
M. Flathers, Phuong Anh Nguyen, J. Noorily et al.· 0 citations
A multimodal emotion-aware architecture, which pays attention to memory-enhanced personalization and emotion-specific reinforcement learning, is introduced and hybrid human-AI approaches, which focus on safety and empathetic conversation to improve current mental health systems are recommended.
Rainn Ji· SHS Web of Conferences· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
