Guideline adherence and expert-rated potential safety concerns in AI-generated dietary advice for patients with cirrhosis: a comparative evaluation of large language models

Abstract

Background Patients with cirrhosis require complication-specific nutritional counseling. Large language models are increasingly used for health information seeking, but the guideline adherence and potential safety concerns of their dietary advice for cirrhosis remain uncertain. Methods Thirty-six standardized simulated cases represented compensated cirrhosis, cirrhosis with ascites, and hepatic encephalopathy recovery. Each case was submitted once to DeepSeek, ChatGPT, Qwen, and Claude through their official web interfaces, generating 144 English-language recommendations. Three reviewers performed de novo model-blinded assessments using a locked framework. The primary outcome was response-level Guideline Adherence; secondary outcomes included Error Count, Any Major Error, and Harm Flag. Results Mean Guideline Adherence was 7.00 ± 0.00 for ChatGPT, 6.97 ± 0.17 for Claude, 6.94 ± 0.19 for DeepSeek, and 6.44 ± 0.84 for Qwen (Friedman p < 0.001). In hepatic encephalopathy recovery, Qwen scored 5.36 ± 0.54 and was lower than each other model after Holm adjustment (all adjusted p = 0.009). Any Major Error occurred in 0 ChatGPT, 2 Claude, 4 DeepSeek, and 11 Qwen responses; corresponding Harm Flag counts were 0, 1, 2, and 8. Most events occurred during hepatic encephalopathy recovery and involved protein-related recommendations. Conclusion Under standardized English-language structured prompting, all four models generally reproduced core cirrhosis nutrition principles. However, Qwen showed lower adherence and more frequent protein-related errors in hepatic encephalopathy recovery. Expert verification remains necessary before patient-facing or individualized use.