FragBERTa: a fragment-aware molecular representation model with sequential attachment-based fragment embeddings

Abstract

Automated exploration of chemical space through machine learning has become a central strategy in modern drug discovery, where the ability to learn transferable molecular representations directly impacts predictive modeling performance. Recent advances in chemical language models have primarily relied on atom-level string encodings such as SMILES or SELFIES, treating molecules as sequences of individual atoms or characters. However, medicinal chemistry reasoning is often fragment-centric, emphasizing scaffolds, functional groups, and modular substructures rather than isolated atoms. In this study, we introduce FragBERTa, a molecular fragment-aware transformer-based representation language model pretrained using masked language modeling on Sequential Attachment-based Fragment Embedding (SAFE) representations. SAFE decomposes molecules into chemically meaningful fragments while preserving attachment information, enabling the model to learn fragment-level structural patterns within a sequence framework. FragBERTa is pretrained on 11 million unique SAFE sequences extracted from multiple public chemical databases and subsequently finetuned on diverse MoleculeNet benchmarks covering both regression (ESOL, FreeSolv, Lipophilicity, PDBbind) and classification (BACE, BBBP, HIV, Tox21, SIDER) tasks under both random and scaffold splits. Our evaluation shows that FragBERTa achieves competitive performance across tasks, often matching or surpassing prior models. In regression, it demonstrates strong gains in lipophilicity and the best scaffold-split performance on protein-ligand binding affinity within our comparison. In classification, FragBERTa attains the strongest reported random-split ROC-AUC results on BACE, HIV, and Tox21 and the strongest reported scaffold-split ROC-AUC performance on BBBP among the compared baselines, and delivers robust performance across bioactivity, permeability, and toxicity benchmarks, indicating that fragment-based representations effectively capture biologically driven and interaction-sensitive endpoints. Collectively, these findings suggest that fragment-based string representations offer advantages over atom-level representations such as SMILES and SELFIES for scaffold-sensitive and interaction-driven tasks, highlighting the value of fragment-level inductive bias for future molecular foundation models in drug discovery.