Benchmarking large language models against practicing clinicians on psychopathological assessment

Advancing Psychiatric Diagnosis with Large Language Models: Interpretability, Prompting Strategies and Clinical Applications

: Large Language Models (LLMs) have shown potential to improve psychiatric assessment by addressing limitations in traditional diagnostic methods. This review summarizes recent developments in applying LLMs to mental health evaluation, focusing on three core strategies: questionnaire emulation (e.g., GPT-based PHQ-9/GAD-7), free-text classification of clinical narratives and social media posts, and integrated diagnostic-treatment workflows based on DSM/ICD criteria. Emphasis is placed on prompt engineering techniques — such as chain-of-thought and diagnostic reasoning prompts — that enhance model interpretability by generating stepwise rationales. These methods allow LLMs to mimic clinical reasoning while producing transparent, structured outputs. Empirical studies report high internal consistency and moderate-to-strong agreement with validated tools, along with performance metrics that approach or surpass human baselines in selected tasks. Key challenges include generalization across cultural contexts, explanation fidelity, and clinical applicability. Addressing these issues will require robust prompt design, alignment with clinical guidelines, and validation in real-world settings. This review provides a framework for understanding LLM-based diagnostic methods and outlines directions for future development in computational psychiatry.

Scalable, context-sensitive psychiatric assessment with large language models and brief diaries

It is shown that LLMs can assess most major forms of psychopathology from mere minutes of audio, and these results support scoring open-ended narratives with LLMs as a scalable, portable method to translate idiographic diagnostic data into standardized psychiatric assessments.

Performance of a domain-specific large language model in answering patient questions in psychiatry

MIND was able to answer many questions about escitalopram in a manner deemed accurate, complete, and safe by psychiatrists the majority of the time, and represents a step towards building safe LLM systems to enhance patient education in psychiatry.

Research protocol: Evaluating Chain-of-Thought Reasoning in LLMs for Complex Clinical Psychiatric Cases

Background Psychiatric medicine presents unique diagnostic and therapeutic challenges, often involving multimorbidity, polypharmacy, and atypical presentations requiring complex reasoning. Artificial Intelligence, particularly Large Language Models (LLMs), is emerging as a support tool in these settings. However, the clinical validity, interpretability, and reliability of LLMs in psychiatry remain largely unexplored, particularly their ability to generate transparent, guideline-consistent reasoning. Aims This study evaluates the clinical reasoning capabilities of LLMs in complex psychiatric scenarios. The primary aim is to assess the validity of their long chain-of-thought (CoT) reasoning. A secondary aim is to determine whether LLM assistance improves clinicians’ diagnostic accuracy. Method Three LLMs, Gemini 2.5, Grok, and DeepSeek R1, will be assessed using ten complex psychiatric cases sourced from a non-public clinical manual and rated with the Amsterdam Clinical Challenge Scale. Each model's CoT response will be evaluated by blinded panels of psychiatrists, residents, and general practitioners using standardized metrics for factual accuracy, coherence, and medical plausibility. In a second phase, clinicians will answer thirty diagnostic questions with and without support from the best-performing LLM. The study uses step-by-step reasoning prompts and few-shot examples to elicit detailed responses and includes bias mitigation strategies such as randomization, blinding, and statistical controls. Results Analyses will assess inter-rater reliability, metric redundancy, and reasoning quality. Closed-source models are expected to outperform open-source ones. LLM assistance is anticipated to improve diagnostic accuracy, especially among non-specialists. Conclusions This study provides a framework for evaluating LLMs in psychiatry and supporting their safe, evidence-based integration into mental health practice.

Language Models Reproduce Human Reductionist Bias and Decision Inconsistency in Neurodevelopmental Disorders Assessment

Large language models (LLMs) are increasingly supporting complex mental-health decisions, which depend not only on factual evidence but also value-laden interpretations. We introduce a mixed-methods human-LLM auditing framework examining decision consistency, susceptibility to cognitive heuristics, declarative intellectual humility, and the concepts operationalized in support-allocation judgments of neurodevelopmental disorders. Comparing 35 humans (18 physicians and 17 psychologists) with seven LLMs, we show that in both groups, ratings of patients'functional level were not significantly associated with support-eligibility decisions, indicating an inconsistency between descriptive assessments and final evaluative judgments. Specifically, we find that neither group showed significant susceptibility to experimental manipulations targeting anchoring and representativeness heuristics. LLMs reported higher intellectual humility than experts (U = 241, p<.001, r = .62; LLMs: M = 41.43, SD = 1.99; experts: M = 29.03, SD = 8.05), but it was unrelated to decision consistency or functional assessment. While LLMs and physicians granted support less frequently than psychologists (U = 180.50, p = .003, r = .34), they also interpreted a concept of"basic life needs"differently, primarily as biological survival and self-care, and not communicative and social needs. These findings suggest that despite expressing high levels of intellectual humility, LLMs reproduce a reductionist interpretive framework and knowledge embedded in medical decision-making. More broadly, we argue that evaluating AI in high-stakes contexts requires not only measuring accuracy, agreement, or resistance to cognitive bias, but also critical examination of the concepts of neurodiversity that AI systems operationalize.

Epistemic injustice and phenomenological reductionism in psychiatric AI: an empirical-philosophical analysis