Multimodal LLMs for Leprosy Diagnostic Support: Evaluating Performance and Healthcare Professionals’ Trust and Acceptance

Abstract

Leprosy remains a major public health challenge in high-burden regions, where its clinical similarity to other dermatoses and reliance on clinical assessment can contribute to delayed diagnosis. Although multimodal Large Language Models (LLMs) have shown potential for dermatological decision support, their performance for leprosy and their acceptance by healthcare professionals remain largely unexplored. This study addressed these issues through two complementary experiments. First, four general-purpose LLMs (ChatGPT, Grok, Claude, and Gemini) were evaluated on 213 clinical cases across three input modalities (image, metadata, and combined image-and-metadata) and four prompting strategies (zero-shot, few-shot, chain-of-thought, and prompt repetition). Second, 62 healthcare professionals were randomised to evaluate LLM-generated diagnostic responses with or without an accompanying Chain-of-Thought (CoT)-generated rationale, assessing trust, perceived usefulness, and intention to use. The best-performing configuration achieved an F1-score of 0.837. Structured clinical metadata provided greater discriminative value than images alone, while combining images with metadata did not consistently improve performance beyond metadata alone. Performance also varied according to the interaction between model, input modality, and prompting strategy, with no prompting strategy being uniformly superior. In the human-centred evaluation, trust did not differ significantly between experimental conditions (p = 0.469), whereas perceived usefulness was higher in the group when diagnostic conclusions were accompanied by CoT-generated rationales (p = 0.043); intention to use showed a similar direction but did not reach statistical significance (p = 0.082). Overall, general-purpose LLMs show potential for leprosy diagnostic support, but their clinical evaluation should extend beyond aggregate classification performance to consider clinically informative inputs, error profiles, response consistency, and the presentation of outputs in ways that support appropriate human oversight.