Understanding Structural Representation in Foundation Models for Polymers

Abstract

From the relative scarcity of training data to the lack of standardized benchmarks, the creation of effective foundation models for polymers faces significant and multi-faceted challenges. At the core, many of these issues are tied directly to the structural representation of polymers. Here, we present a chemical language foundation model built on using a SMILES-based polymer graph representation (CPG) that incorporates polymer architectural features and connectivity that are often missing in other line notations. This foundation model exhibited excellent performance on 30 different polymer property benchmark datasets. Critical evaluation of the developed representation against other variations in control experiments reveals this approach to be a robust method of representing polymers in language-based foundation models. These experiments also reveal a strong invariance of structural representations to small perturbations, with many variations of structural representation exceeding or equaling state-of-the-art (SOTA) performance. Surprisingly, SMILES representations which are chemically or semantically invalid also provided near or SOTA performance in several instances--underscoring an unexamined blind spot in the development of chemistry language models. Examination of error sources and attention maps for the evaluated structural representations corroborate the findings of the control experiments, highlighting the ability of the model to interpolate SMILES sequence space in a manner that is loosely congruent to chemical and architectural space for polymers. Overall, this work highlights the surprising robustness of chemistry language models to structural representation perturbations and identifies the conditions under which CPG representation provides meaningful advantages.