Shedding light on neural learning to rank models for anticancer drug prioritization

Abstract

Learning to Rank (LeToR) methods have gained increasing attention in drug response prediction, offering a direct way to prioritize effective treatments for cancer cell lines. In this study, we systematically benchmark six ranking loss functions, including state-of-the-art listwise methods, and five types of molecular representations across two large-scale drug screening datasets, CTRP and PRISM. Using high-dimensional gene expression profiles and various drug fingerprints and descriptors, we evaluated models under multiple validation setups and ranking metrics. Our results demonstrate that listwise loss functions such as LambdaLoss and LambdaRank consistently excel in both early and overall ranking quality. Additionally, combining molecular fingerprints with physicochemical descriptors yielded improved performance. A novel attention-based mechanism and a modified version of RankingSHAP were integrated to enhance interpretability, uncovering key genes and substructures aligned with known biological insights. The explainability pipeline successfully distinguished estrogen receptor-positive (ER⁺) and estrogen receptor-negative (ER−) breast cancer subtypes. The model successfully identified critical substructures in docetaxel, an FDA-approved therapy, and triptolide, which is currently undergoing clinical evaluation for breast cancer. These findings are consistent with established structure-activity relationship (SAR) data. Overall, this study presents a comprehensive evaluation framework and underscores the importance of carefully selecting loss functions and feature representations when developing robust and interpretable drug-ranking systems.