Transformer-based molecular fragment prediction using SMILES and DeepSMILES representations in a fragment-based drug discovery pipeline
Fragment-based drug discovery (FBDD) uses small molecular fragments as starting points for drug development, and machine-learning models that operate on molecular string representations are increasingly applied to fragment-related tasks. SMILES is the dominant such representation, but its paired ring closures and balanced parentheses introduce syntactic complexity that can affect model behavior. We present a controlled study of how molecular string representation influences transformer-based fragment recovery, using an FBDD-motivated label pipeline: reference fragments are derived from known drugs via RECAP fragmentation and docking-based ranking, and the model is scored on recovering them. We introduce DeepBERTa, a ChemBERTa-derived transformer pretrained on DeepSMILES, and evaluate it against a matched SMILES baseline under identical architecture, data splits, and optimization (approximately 34,000 drug–fragment pairs). DeepSMILES produces syntactically valid predictions more often than SMILES (54.2% vs. 43.5%) and a higher full test-set mean Tanimoto similarity (0.36 vs. 0.29). A per-sample selection between the two representations raises mean Tanimoto to 0.43, and a deployable variant that selects on model confidence rather than the reference recovers most of this gain. Among molecules where one representation strictly wins, DeepSMILES wins more often than SMILES (27.8% vs. 16.3% of test molecules). These results show that string representation meaningfully influences transformer fragment recovery and that representation-aware selection outperforms either representation alone.