DICD-IE: an expert-annotated benchmark for evaluating pretrained biomedical language models in dental implant literature and clinical knowledge extraction

Abstract

Dental implant literature contains clinically important information on risk factors, treatments, outcomes, and, in rare cases, co-occurrences with oral cancer have been reported. However, this information is typically embedded in concise narrative statements, which complicates automated extraction and limits large-scale evidence synthesis and structured clinical information extraction. Because existing biomedical datasets and pretrained models are insufficient for this domain due to a lack of implant-specific terminology, procedural context, and clinically relevant relationships reported in implant literature, a new approach is needed. To address this challenge, we developed DICD-IE (Dental Implant Clinical Dataset for Information Extraction) as an expert-annotated corpus for Named Entity Recognition (NER) and Relation Extraction (RE), which includes an exploratory oncologic dimension based on nine rare case-report articles. Specifically, the dataset comprises 2,204 clinically meaningful text segments from 30 peer-reviewed full-text articles, including 16 entity types and 21 relation types. Five domain experts annotated the corpus using a consensus-based protocol, achieving strong inter-annotator agreement across tasks, with overall agreement, average pairwise agreement, and Gwet’s AC1 values of 0.8499, 0.8936, and 0.8624 for NER and 0.7808, 0.8622, and 0.8577 for RE, respectively. Six biomedical transformer models were evaluated using the dataset. SciBERT achieved the best performance under the sentence-level benchmark, with F1 scores of 0.6213 for NER and 0.6274 for RE. Additional article-level evaluation, designed to reduce article-specific information overlap between data splits, produced more conservative estimates (F1 = 0.6041 for NER and 0.5162 for RE), highlighting the importance of stricter validation strategies for domain-specific biomedical NLP datasets. Furthermore, cross-dataset evaluation using two protocols, forward and reverse transfer, provided exploratory insights into model transfer behavior across biomedical datasets and highlighted the challenges posed by domain-specific terminology, annotation differences, and implant-specific clinical narratives. The oncologic component of DICD-IE provides contextual representation of reported co-occurrences rather than evidence of causal relationships or clinical prediction. Knowledge graph construction further demonstrated the potential of the dataset for organizing implant-related clinical concepts and relationships. Overall, DICD-IE provides a foundation for dental implant information extraction, knowledge graph development, and future document-level clinical NLP research.