Does a general-purpose large language model improve physicians’ clinical reasoning?

Abstract

This paper investigates whether access to a general-purpose large language model (LLM) improves physicians’ clinical reasoning across diverse healthcare contexts, as well as the possible implications of using an LLM in healthcare settings. Using a randomized controlled trial with 249 physicians in Indonesia, Kenya, and the Netherlands, the study finds that LLM access enhances performance on standardized clinical vignettes in all three countries. The magnitude of improvement varies, with the largest gains observed in Kenya (+18%), followed by Indonesia (+10.7%) and the Netherlands (+7.2%).The results, however, reveal substantial heterogeneity. Performance distributions overlap, and some physicians with LLM access perform worse than those without, indicating that access alone does not guarantee improvement. Higher usage is associated with better outcomes, and less specialized physicians appear to benefit more, implying that LLMs may help reduce skill gaps. Importantly, the findings emphasize that LLMs function as complements rather than substitutes for clinical expertise. However, the study identifies important risks, including automation bias, hallucinations, and context misalignment, underscoring the importance of careful integration, training, and governance.The paper concludes that while LLMs can enhance clinical reasoning, their effectiveness depends critically on how they are implemented within healthcare systems. The paper recommends that policymakers prioritize structured integration of LLMs as decision-support tools, combined with targeted training, local validation, and safeguards against automation bias rather than relying on access alone. It also emphasizes the need for investment in infrastructure, continuous monitoring, clear liability frameworks, and inclusive governance to ensure equitable, safe, and context-appropriate deployment. It argues for the importance of social dialogue in managing the process.