Transformer models have revolutionized natural language processing (NLP), and text-based molecular representations like SMILES have successfully extended these architectures to chemistry. However, domain-adaptive pre-training often causes models to overfit to chemical syntax, catastrophically forgetting their foundational semantic capabilities. To address this challenge, we introduce CheMatE, a chemistry-oriented embedding model that jointly captures molecular structure and domain-specific natural language within the same representation space. Built on a ModernBERT backbone, CheMatE learns bi-semantic representations through a two-stage training procedure: continued masked language modeling (MLM) followed by a Matryoshka contrastive learning stage via Multiple Negative Ranking Loss (MNRL). First, we train the model using MLM on a novel, large-scale corpus of SMILES-annotated, long-context scientific documents that were constructed and curated from FineWeb and ChemPile (comprising 10.4B and 11.5B tokens, respectively). Subsequently, the model undergoes contrastive learning using a synthetic dataset of SMILES-text pairs algorithmically derived from our original training corpus. This design exposes the model to SMILES-enriched scientific literature, enabling bi-semantic understanding. We evaluate CheMatE across a range of downstream tasks covering molecular property prediction and scientific language understanding. Our results demonstrate that coupling our custom-curated datasets with this sequential training strategy yields robust, highly transferable representations. By effectively unifying structural and contextual signals within a single text-based framework, CheMatE achieves competitive performance across both specialized chemistry models and general-purpose language model baselines.
Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.
Large language models (LLMs) provide a promising direction for learning molecular representations from text-like inputs, yet most molecular contrastive learning methods still rely on graph- or SMILES-level augmentations that may unintentionally distort chemical structure. We propose MolPACL, a prompt-augmentation-based supervised contrastive learning framework that incorporates high-level chemical semantics while preserving molecular identity. MolPACL generates multiple semantically consistent prompt views for each molecule from its SMILES string and physicochemical descriptors using diverse templates and lightweight lexical perturbations. These views are combined with task-aware class positives and negatives to form contrastive batches, and the model is trained using a supervised objective based on the Soft Nearest Neighbor loss. Experiments on MoleculeNet benchmarks show that the proposed approach achieves strong performance on both classification and regression tasks while reducing training cost, requiring no additional molecular pretraining and using a relatively small pretrained LLM.
Ali Forooghi, Luis Rueda, A. Ngom· IEEE transactions on computa...· 0 citations
MolLingua, a token-centric, dual-modal framework designed for native molecular understanding, uses a dual-branch Residual Vector Quantization engine to discretize heterogeneous, high-dimensional spatial 2D and 3D features into compact code sequences rather than relying solely on continuous projections.
Haoyang Liu, Xikang Feng, Fei Guo et al.· IEEE journal of biomedical a...· 0 citations
Automated exploration of chemical space through machine learning has become a central strategy in modern drug discovery, where the ability to learn transferable molecular representations directly impacts predictive modeling performance. Recent advances in chemical language models have primarily relied on atom-level string encodings such as SMILES or SELFIES, treating molecules as sequences of individual atoms or characters. However, medicinal chemistry reasoning is often fragment-centric, emphasizing scaffolds, functional groups, and modular substructures rather than isolated atoms. In this study, we introduce FragBERTa, a molecular fragment-aware transformer-based representation language model pretrained using masked language modeling on Sequential Attachment-based Fragment Embedding (SAFE) representations. SAFE decomposes molecules into chemically meaningful fragments while preserving attachment information, enabling the model to learn fragment-level structural patterns within a sequence framework. FragBERTa is pretrained on 11 million unique SAFE sequences extracted from multiple public chemical databases and subsequently finetuned on diverse MoleculeNet benchmarks covering both regression (ESOL, FreeSolv, Lipophilicity, PDBbind) and classification (BACE, BBBP, HIV, Tox21, SIDER) tasks under both random and scaffold splits. Our evaluation shows that FragBERTa achieves competitive performance across tasks, often matching or surpassing prior models. In regression, it demonstrates strong gains in lipophilicity and the best scaffold-split performance on protein-ligand binding affinity within our comparison. In classification, FragBERTa attains the strongest reported random-split ROC-AUC results on BACE, HIV, and Tox21 and the strongest reported scaffold-split ROC-AUC performance on BBBP among the compared baselines, and delivers robust performance across bioactivity, permeability, and toxicity benchmarks, indicating that fragment-based representations effectively capture biologically driven and interaction-sensitive endpoints. Collectively, these findings suggest that fragment-based string representations offer advantages over atom-level representations such as SMILES and SELFIES for scaffold-sensitive and interaction-driven tasks, highlighting the value of fragment-level inductive bias for future molecular foundation models in drug discovery.
HME is presented, a framework that combines multiple views of molecules to improve molecular understanding and design and enables bidirectional navigation of the chemical-linguistic space, achieving consistent improvements across molecular comprehension and design tasks over strong baselines.
Liuzhenghao Lv, Hao Li, Yu Wang et al.· Nature Communications· 0 citations
Large language models (LLMs) are widely applied across chemical tasks, such as molecular property prediction, which underpins drug discovery. Molecular LLMs represent a molecule through several modalities, notably a 1D SMILES sequence or a 2D molecular graph. Both encode molecular information implicitly, so the contribution of individual substructures remains opaque. Retrieval and augmentation methods add context, but from external sources. However, the cues chemists reason over are the internal substructures that drive a property up or down. We propose MR-MoL, a multi-granular rationale-guided molecular LLM that supplies this evidence directly. A fine-tuned GNN scores each substructure through masking, and the most influential ones are serialized as a ranked, direction-tagged rationale that the LLM reads alongside the SMILES sequence and molecular graph. The rationale spans three levels of granularity: Murcko scaffolds with their side chains, BRICS fragments, and functional groups. This is, to our knowledge, the first method to expose GNN-derived attributions to an LLM as evidence for property prediction. On eight MoleculeNet tasks, MR-MoL achieves the best overall results among generalist models and narrows the gap to specialist models tuned for each task. Five diagnostics further confirm that the model reads the rationale rather than merely benefiting from its presence. Its direction, rank, and substructure each shape the prediction, and its attributions reproduce known structure-property relationships.
Junwoo Park, Minyoung Shin, C. Lee et al.· 0 citations