Validated Semantic Scoring Improves Filtering of Generic Concepts from Knowledge Graphs for Drug Repurposing Queries

Abstract

When an ontology-based biomedical knowledge graph (a KG, containing information about genes, pathways, phenotypes, and molecular interactions) is analyzed using a computational reasoning system, the system can return unwanted generic concepts. Such generic concepts, in the context of a query about potential drug treatments for a disease, might include ‘pharmaceutical preparations’ or ‘antibiotics’. Answers can be filtered based on hand-curated block-lists or ontology Information Content (IC), but to our knowledge, (1) neither filtering method has been validated against expert judgments of generality, and (2) IC is not available for the majority of nodes in the representative KG used in this work. In this study, three domain experts rated representative lists of KG concepts for generality. We built a semantic precision score (SPS) from a concept’s name and description alone, using a biomedical language model (BioLORD). For the purpose of filtering generic answers from biomedical query results, we compared (using queries for 200 diseases) the SPS against a curated concept block-list and two IC thresholds. IC agreed moderately with the experts (Spearman’s ρ = 0.59), but correlations were negligible or negative within IC bands. The SPS was well-correlated with the experts (ρ = 0.76) globally and within SPS bands, and remained informative on concepts without IC (ρ = 0.59). Hard filtering of query results using the SPS removed 1,502 of 33,161 query answers, nearly three times the block-list’s 513, while removing fewer of the query-specific DrugCentral indications. Hard filtering by IC removed either few answers or many approved drugs. We additionally compared the filtering methods’ performance on an independent evaluation set of queries for 200 diseases. The SPS filter removed 5,135 of 91,203 answers compared with the block-list’s 1,419 and lost none of 1,296 indications compared with 17. Further, SPS filtering removed approved drugs no more often from the half of DrugCentral not used in training than from the half used in training. Semantic meaning captures expert judgments of specificity more faithfully than ontology position, and semantic scoring can replace IC and manually maintained block-lists for filtering drug-repurposing results.