Aug 2026· Journal of Biomedical Informatics· pp.
105093
· 0 citations· 60 references
Medicine
TL;DR
This study finds that existing MCN benchmarks present data quality issues and underexplored data fusion potential, and data quality enhancement and LLM-based controlled-variety data augmentation help alleviate overlapping phrases and long-tail issues.
Abstract
Objective
Medical Concept Normalization (MCN) maps informal health phrases to formal clinical concepts. It is a critical task for pharmacovigilance, patient record analysis, and health-related text mining. Extensive MCN research has mainly relied on single-source datasets and overlooked data quality (DQ) issues. This study aims to develop a quality-aware data fusion framework for MCN using Large Language Models (LLMs).
Methods
The methods consist of DQ evaluation and enhancement, LLM-based data augmentation, and multi-source MCN dataset fusion using enriched concept-phrase pairs. We analyze six widely used MCN datasets-AskAPatient, CADEC, COMETA, PsyTAR, TwADR-S, and TwiMed-each collected from social media and mapped to SNOMED-CT. Our evaluation metrics include correctness, concept validity, coverage, semantic variety, and class imbalance. For DQ enhancement and data augmentation, we use Gemini for zero-shot and few-shot learning to increase the semantic variety and phrase count for rare concepts. We then perform data fusion based on shared medical concepts.
Results
The DQ evaluation reveals substantial issues, including incorrect mappings, invalid concepts, low-variety redundant phrases, and long-tail concept-phrase distribution. After augmentation, phrase counts increase by 172.1%, and by 450.0% after fusion. To directly investigate model performance improvement on rare cases, we introduce concept-level macro metrics. SapBERT, KNN-BioEL, and KRISSBERT trained on the augmented and fused dataset achieve significant gains in accuracy, recall, precision, and F1 over single-source baselines, up to 38%.
Conclusion
Our study finds that existing MCN benchmarks present data quality issues and underexplored data fusion potential. Data quality enhancement and LLM-based controlled-variety data augmentation help alleviate overlapping phrases and long-tail issues. Moreover, quality-aware data fusion can expand conceptual coverage, improving MCN performance. This work highlights data quality evaluation and fusion strategies are effective in advancing MCN. We hope these contributions could support quality-guided large-scale MCN data generation with minimal label costs, strengthen reliable biomedical text mining and downstream applications.
AVAILABILITY
https://github.com/yhZHOU515/DataFusion4MCN.
The proposed framework offers a practical and scalable approach to mitigating hallucinations without requiring task-specific fine-tuning, highlighting the potential of retrieval-augmented approaches for trustworthy artificial intelligence (AI)-assisted healthcare applications.
This study evaluated both closed-source and open-source large language models for extracting and prioritizing medical jargon from EHR notes relevant to individual patients, leveraging prompting techniques, fine-tuning, and data augmentation and found that model performance could deviate largely based on prompting styles.
W. Jang, Sharmin Sultana, Zonghai Yao et al.· JMIR AI· 1 citation
This is the first study to benchmark SLMs on Italian EHRs and investigate the role of clinical expertise in prompt engineering, offering valuable insights for the future integration of SLMs into real-world clinical workflows.
Federica Corso, V. Peppoloni, L. Mazzeo et al.· Communications Medicine· 0 citations
Translating complex biomedical data into patient-friendly narratives is central to modern biomedical informatics. This study presents a comparative analysis of training small language models (SLMs) in specialized biomedical datato-text generation tasks. We explore widely adopted post-training methods including supervised fine-tuning (SFT), direct preference optimization (DPO), odds ratio preference optimization (ORPO), and group relative policy optimization (GRPO) with Qwen-based SLMs on a medicine package leaflets dataset. To assess cross-dataset generalizability, we also curated drug label data from openFDA. We evaluate models using both standard lexical overlap metrics like ROUGE as well as semantic similarity measures. Across our experiments, the results show that (1) the aligned SLMs outperform proprietary models like GPT-5; (2) ORPO outperforms the SFTbaselines; (3) GRPO yields the most robust cross-dataset performance among the alignment methods tested as well as GPT-5.
Xi Yang, Guodong Liu, Chuqin Li et al.· 0 citations
The findings support the feasibility of applying LLM-based natural language processing tools in resource-limited, non-English healthcare settings and should assess emerging high-parameter models and explore additional clinical domains.
Breno Gabriel Araújo Sampaio de Jesus, Tomaz Castrillon Figueiredo, Clariele de Almeida Pereira et al.· Cadernos de Saúde Pública· 1 citation
Background: Korean-language medical question-answering datasets are increasingly used for large language model (LLM) development, but structural completeness alone does not establish model performance or clinical validity. We examined two national Korean medical knowledge datasets by combining a public labeling-file audit with a secondary evaluation of item-level outputs distributed with the corresponding official LLM packages. Methods: We reviewed the official documentation and complete public download inventories, parsed 34 training/validation labeling archives containing 31,036 records, and audited required fields, duplication, text length, question type, and specialty distribution. We also audited the official Qwen2.5-14B LoRA model packages and independently recalculated performance from their distributed item-level KorMedMCQA output files (2494 identical items per model). Accuracy was reported with Wilson 95% confidence intervals; model outputs were compared using an exact McNemar test and a paired bootstrap confidence interval. Because no compatible local GPU was available, model inference was not independently rerun. Results: The documentation described 34,487 labeled QA pairs, of which 31,036 (90.0%) were present in the publicly accessible training/validation labeling files; no public test-labeling archive was listed. Required fields were complete, no duplicated qa_id values were found, and one duplicated question-answer pair occurred in the Essential dataset. Multiple-choice items comprised 78.7% of all documented QA pairs, and the top three domains comprised 56.8%. In the distributed KorMedMCQA outputs, the Essential-care model answered 1603/2494 items correctly (64.27%; Wilson 95% CI 62.37–66.13%), while the Specialized-medicine model answered 1596/2494 correctly (63.99%; 95% CI 62.09–65.85%). The paired difference was 0.28 percentage points (bootstrap 95% CI −0.44 to 1.00), with no significant difference by exact McNemar test (46 vs. 39 discordant correct items; p = 0.515). Conclusions: The public training/validation labeling files showed favorable basic structural completeness, but the unavailable test labels, multiple-choice predominance, domain imbalance, limited record-level provenance, and incomplete model-package traceability constrain claims of clinical readiness. The distributed model outputs demonstrated moderate examination-style benchmark performance without a significant difference between the two models. These findings support use as research infrastructure, not evidence of clinical validity, and indicate the need for independent inference reproduction, clinician-led open-ended and safety evaluation, temporal updating, specialty-stratified reporting, and human oversight before clinical use.
