A novel benchmark comprising over 9,000 real-world, point-of-care, multilingual, and multimodal clinical question-answer pairs sourced from frontline health workers in Nigeria reveals several critical insights into the suitability of LLMs as clinical decision support systems in low-resource contexts.
Abstract
In this study, we introduce a novel benchmark comprising over 9,000 real-world, point-of-care, multilingual, and multimodal clinical question-answer pairs sourced from frontline health workers in Nigeria. Using the dataset, we compare local general practitioners to multiple leading open and closed LLMs. Our results reveal several critical insights into the suitability of LLMs as clinical decision support systems in low-resource contexts. The results confirm that performance varies widely by language and input modality (e.g., text vs speech): while models perform best on English text inputs, their accuracy drops significantly for local-language speech. Critically, it is possible to achieve substantial performance gains by transcribing and translating other languages into English before prompting an LLM-- an important insight for non-anglophone product developers. Finally, this benchmark highlights key limitations of SLMs in supporting frontline healthcare in low-resource settings and provides a clear opportunity to track improvements as novel solutions are developed.
VITA's advantages in accuracy and completeness persisted under the neutral judge; its communication scores were lower, and this results indicate that a purpose-built clinical RAG system remains competitive with frontier LLMs on an open benchmark, consistent with corpus specificity as a design variable that improves grounding at some cost to communication polish.
Praveen Reddy, C. Mandke, Suvrankar Datta et al.· 0 citations
Safety evaluation of large language models is conducted predominantly in English and predominantly on frontier systems. Neither condition describes how such models are encountered in low-resource health settings, where small quantised systems are run locally and queried in local languages. We ask whether clinical safety established in English transfers to Hausa, and whether any failure is attributable to the language, the clinical task, or the class of model that low-resource deployment admits. Matched English-Hausa question pairs were built for three conditions of high burden in northern Nigeria: malaria, sickle cell disease, and tuberculosis, probing knowledge recall, emergency triage, a leading question inviting a contraindicated action, and a traditional-remedy claim. Six models were evaluated: five locally deployable systems of 4-9 billion parameters, two medically fine-tuned, and one frontier system. All 128 responses were scored against Nigerian national treatment guidelines by two fluent Hausa speakers working independently and blind to one another. Among locally deployable models, mean clinical correctness fell from 1.57 in English to -0.03 in Hausa, on a scale where 2 denotes a correct answer and -1 an actively harmful one. The frontier model moved from 2.00 to 1.75 and produced no response judged harmful in either language. Drift was consistent across all three conditions. Inter-rater agreement was substantial for clinical correctness (kappa = 0.70); agreement on harm was initially poor (kappa = 0.22) and is examined in detail. Because a frontier model answers the same questions competently in Hausa, the deficit is a property neither of the language nor of the clinical material, but of the deployable tier.
Anthonio Oladimeji Gabriel, Dimeji AbdulSobur Olawuyi, T. Ajayi et al.· 0 citations
It is suggested that domain-adapted encoder models may be preferable for similar structured clinical NER settings, although larger and externally validated benchmarks are needed before generalizing to other languages, clinical corpora, model families, or deployment environments.
L. Elvas, Carolina Carvalho· Scientific Reports· 0 citations
In the rapidly advancing field of AI-driven telehealth services, effective medical communication in India remains a significant challenge due to the language barrier, as most of the population is not proficient in English. Additionally, many patients struggle to accurately describe their medical conditions using text alone. As a result, an essential feature of any telehealth service is the ability to supplement textual queries with medical images, enabling doctors to conduct a more careful analysis and provide well-informed diagnoses and treatment recommendations. In this work, we introduce IndicMedQA , a novel multimodal AI framework that integrates Indic large language models (LLMs) and visual encoders to analyze patient inquiries using both textual and visual cues. To support this, we create a multilingual multimodal medical corpus spanning seven major Indian languages, translated using a semi-automated approach. This dataset facilitates medical understanding for every input query and its associated medical image—the output is a detailed patient summary, symptom analysis, probable conditions, additional findings, and severity assessment with medical precision. Our framework significantly enhances personalized healthcare experiences, ensuring context-aware multimodal understanding of patient needs in Indic languages. Extensive experiments demonstrate that IndicMedQA surpasses all baselines, establishing a new benchmark for Indic AI in healthcare. Disclaimer: This work contains medical images that depict the subject matter of the study, which may be disturbing to some readers.
Akash Ghosh, Arkadeep Acharya, M. Muhsin et al.· ACM Transactions on Computin...· 0 citations
ObjectiveLarge language models (LLMs) have the potential to revolutionize how population-level health research is conducted by automatically abstracting data that would otherwise be unavailable. However, few results are available on real clinical notes. We developed an LLM leaderboard showing how open-source LLMs perform at entity extraction on unseen clinical notes.
ApproachEMR data, including free-text notes, were linked to a chart-review cohort comprising 10,659 adults admitted to a hospital in Calgary, Canada, between 2017 and 2022, with data on comorbidities. We then attempted to replicate this chart review with multiple open-source LLMs in a secure computing environment. Chart review results served as the reference standard.
ResultsThere was a wide variation in performance among the examined LLMs: the smallest, Llama 3.2 3B, had a high mean sensitivity of 0.97 but low PPV at 0.4; the largest, Llama-3-70B, showed a high mean sensitivity of 0.96 and greatly improved PPV of 0.7; in-between these in size, phi 4 demonstrated a more balanced performance with a mean sensitivity of 0.81 and PPV of 0.77. However, the results varied considerably across conditions, with quirks specific to each model.
ConclusionsLLMs are already available that can perform entity extraction well enough to be considered in place of some administrative data. With rapid developments in the field, a leaderboard based on real clinical data is vital for informing researchers on best practices for integrating the latest AI techniques into their data practices.
E. Martin, Seungwon Lee, K. Riazi et al.· International Journal of Pop...· 0 citations
Across medical benchmarks, MedLLM shows a pattern visible only at sub-billion scale: medical competence does not degrade uniformly under compression but splits by task type and dissociation is masked at 7B, where both capabilities are present, and surfaces only when capacity is scarce.
M. R. Rahman, Asim Ahmed, Mihan Mohagheghzadeh et al.· 0 citations