Diagnostic performance of a specific local Turkish large language model in detecting alcohol use disorder across internal medicine practice: A 42-patient pilot study

Abstract

Objective: Alcohol use disorder (AUD) is underdiagnosed in internal medicine. We evaluated a local large language model (LLM), fine-tuned on a Turkish medical corpus, for detecting AUD across outpatient, emergency and inpatient settings. Method: We retrospectively analyzed 42 adult patients at a tertiary state hospital (January–May 2024), in a single-center de-identified dataset modeled on the MIMIC-IV schema. The reference standard was the clinician-confirmed DSM-5 AUD diagnosis; comparators were the Alcohol Use Disorders Identification Test, Consumption version (AUDIT-C), gamma-glutamyl transferase (GGT), mean corpuscular volume and the aspartate aminotransferase/alanine aminotransferase ratio. A Llama-3-8B-Instruct model, fine-tuned via low-rank adaptation on an in-hospital GPU server, produced binary predictions, probabilities and severity categories. Analyses used Wilson 95% confidence intervals, bootstrap AUC, Cohen's κ and the Results: Mean age was 53.0 ± 15.0 years; 50.0% were female. AUD prevalence was 47.6% (20/42). The LLM achieved sensitivity 85.0% (95% CI, 64.0–94.8), specificity 90.9% (72.2–97.5), accuracy 88.1%, area under the curve (AUC) 0.864 (0.720–0.977) and κ = 0.761. Within the 17 true positives, severity matched the reference exactly (quadratic-weighted κ = 1.000), conditional on correct detection in a small subgroup. Within this enriched pilot cohort, AUDIT-C and GGT showed perfect separation (AUC = 1.000) and the LLM AUC was lower (DeLong p = 0.038). This ordering is a ceiling effect of the present cohort and is not a general property of these markers. Conclusion: The local LLM achieved clinically acceptable accuracy and exact severity classification within correctly flagged cases. Its principal value lies where AUDIT-C is not routine...