Unsupervised discovery of functional sequence patterns from protein language model with MotifAE

Abstract

Protein language models (pLMs) learn sequence patterns at evolutionary scale, but these patterns remain inaccessible within these “black box” models. To discover them, we developed MotifAE, an unsupervised framework based on the sparse autoencoder (SAE) architecture that projects pLM embeddings into an interpretable, sparse latent space. MotifAE introduces an additional smoothness loss to encourage coherent feature activation, which markedly improves the identification of known functional motifs compared to the standard SAE. The sequence patterns captured by MotifAE exhibit rich diversity, align with known functional motifs, and are reflected in the model’s weight space. Beyond short motifs, MotifAE also captures some structural domains, with latent feature activation scores correlating with residue importance for diverse domain functions. By aligning MotifAE features with experimental data, we further identified features associated with domain folding stability. These features enable the prediction of a stability-specific fitness landscape. Overall, MotifAE provides a general framework for systematic sequence pattern discovery and interpretation, with the potential to advance protein function analysis, mutation effect interpretation, and rational protein engineering. Protein language models (pLMs) capture biologically meaningful sequence patterns, but the features learned by these models remain largely opaque. Here the authors present MotifAE, an interpretable sparse autoencoder framework that reveals diverse sequence motifs and structural domains from pLM embeddings, improves recovery of known functional motifs and identifies features linked to domain function and folding stability, enabling stability-specific fitness landscape prediction.