Beyond accuracy: evaluating the reliability of large language models for medical assessment

Abstract

Background Large language models (LLMs) perform well on medical examinations, but they are almost always evaluated as test-takers, judged on the accuracy of their answers. Far less is known about their reliability as assessment-automation tools, for instance in extracting examination metadata, or about whether single-run, accuracy-based benchmarks can adequately characterize such tools. Methods Twelve mainstream LLMs (8 domestic, 4 international) were evaluated on extracting eight metadata fields from a 60-item endocrinology examination, each performing the identical task in three independent runs under realistic web-based conditions. Performance was separated into completion rate, conditional accuracy, and an all-fields-correct task success rate (TSR), with cross-run variance treated as a primary outcome. Conditional accuracy and TSR were compared within models, and domestic versus international models were compared on TSR. Results Conditional accuracy was uniformly high, yet TSR was sharply bimodal: five models scored 90% or above and seven below 70%, with none in between. Reliability, not mean accuracy, was decisive. Several models with perfect conditional accuracy collapsed when one or two of their three runs failed entirely, a pattern that single-run evaluation would miss. Failures spanned both model behaviour and platform-level constraints, the latter not removable by prompting. Model origin did not predict performance. On one item, four models independently changed the examiner’s assigned cognitive level to one of their own, a “silent relabeling” with direct implications for item-bank integrity. Conclusion For automated assessment metadata extraction, reliability rather than accuracy determines whether an LLM can be deployed, and national origin is not a meaningful predictor. Such tools should be evaluated by repeated runs reporting cross-run variance, and adopted through task-specific piloting within a workflow that reserves human judgement for semantic and normative fields.