ScrambleBench: a workflow for comparative assessment of structure-based de novo generative models.

Abstract

Generative artificial intelligence (AI) has rapidly advanced over the past decade in the field of drug discovery, particularly for the de novo design of small molecules based on target protein structures. While generative AI has the potential to complement traditional structure-based drug design (SBDD) to discover novel hits, their performance is often assessed using non-standardized evaluation criteria. As the number of generative AI models continue to grow, it becomes increasingly important to determine whether these tools are sufficiently robust and reliable for integration into medicinal chemistry workflow. Here, we propose ScrambleBench, a benchmarking workflow designed to evaluate structure-based generative AI models that align with medicinal chemists' practical objectives to identify chemically diverse, drug-like hit candidates that adopt plausible binding conformations and exhibit favourable docking affinities. Using six representative models (Pocket2Mol, PocketFlow, Lingo3DMol, DiffSBDD, PMDM, and Chemistry42), we systematically assessed performance across diverse target classes, including two GPCRs, two kinases, and two hydrolases. While some generative models show superior performance for particular evaluation endpoints, none demonstrates overall dominance across all evaluated criteria. Notably, despite benchmarked proteins (e.g., CDK2, GSK3β) being present in the training datasets, the models still show limited generalization to target binding sites, which resulted in high redocking RMSD values and low virtual hit rates. Our results highlight the importance of evaluating chemical diversity explicitly and using the recently proposed metrics such as Hamiltonian Diversity (HamDiv) which assess both quantity and dissimilarity of a molecular set. Furthermore, as many generated ligands fail to meaningfully engage the target active site, we propose that future generative frameworks incorporate improved loss functions that place greater emphasis on drug-like physicochemical properties and correct pharmacophore recognition.Scientific contributionWhile numerous de novo generative models have been proposed for structure-based molecular design, objective comparison between methods remains challenging due to inconsistent benchmarking practices and heterogeneous evaluation criteria. ScrambleBench introduces a unified and reproducible benchmarking workflow that integrates diversity analysis, conformational validity assessment, docking reproducibility, pharmacophore matching, and virtual hit rate evaluation within a single framework. By systematically comparing representative generative models using common datasets and standardized assessment criteria, this work advances the field by enabling transparent evaluation model performance and practical applicability. Overall, ScrambleBench provides a holistic medicinal chemistry-oriented framework that identifies methodological strengths, limitations, and opportunities for future model development.