Development of deep learning methods for efficient exploration of ultra-large chemical spaces

Abstract

The exponential growth of synthetically accessible chemical libraries, now exceeding tens of billions of molecules, creates unprecedented opportunities for drug discovery alongside formidable computational challenges. Conventional structure-based virtual screening such as docking, though foundational, struggles to scale to these vast spaces because of prohibitive computational costs. This thesis addresses these limitations with complementary artificial intelligence (AI) methods built on deep and active learning. First, we present Deep Docking Ultra (DDU), an active-learning framework integrating pretrained molecular language models with efficient acquisition. Across 384 virtual screening experiments, DDU achieves up to 45-fold speed-up over prior methods such as Deep Docking, and screens over 10 billion molecules from the Enamine REAL collection in 10 days, a 28,500-fold saving over exhaustive docking. Second, we develop Target Adaptive Reinforcement learning for Sampling Activity landscape (TARSA), combining reinforcement learning with Markov Chain Monte Carlo sampling for ultra-large peptide libraries. Applied to 36 million helical peptides from the Protein Data Bank, TARSA identified 105 synthesis candidates; 15 reduced triple-negative breast cancer (TNBC) cell viability by over 60% at 10 μM, seven were selective for cancer over healthy cells, and three leads reached low-micromolar half-maximal inhibitory concentrations (IC50) against TNBC while remaining non-toxic to peripheral blood mononuclear cells and red blood cells. This is among the largest experimentally validated anticancer-peptide campaigns to date. Third, we introduce Protein Structure Graph-Binding Affinity Regression (PSG-BAR), a graph neural network predicting protein-ligand binding affinity via message passing and cross-attention. Representing proteins and ligands as graphs, it needs no predefined binding site while achieving strong performance and interpretability across diverse protein families. Fourth, we introduce Atomic GFlowNet (A-GFN), a generative framework that builds molecules atom-by-atom rather than from predefined fragments. Through scalable pretraining on offline expert demonstrations from drug-like data, then goal-conditioned finetuning, A-GFN generates novel compounds with high predicted binding affinity while maintaining synthetic feasibility and drug-likeness. Collectively, these methods form a comprehensive toolkit combining graph neural networks, reinforcement learning, active learning, and generative modeling to navigate vast chemical spaces efficiently, showing how algorithmic innovation, with domain knowledge and experimental integration, can overcome longstanding barriers in early-stage drug discovery.