The Words of Proteins: Motif-Level Language Modeling for Interpretable Protein Generation

Abstract

What are the fundamental units of protein sequences? Most protein language models treat amino acids as tokens, yet biological functions are not encoded at the single-residue level. Instead, they emerge from combinations of residues that form functional units that corresponds to conserved sequence motifs. Just like how modern language models work at the level of learned sub-word units instead of characters, we argue that explicitly modeling at the functional motif level provides both mechanistic insight into sequence-function relationships and interpretable control over protein generation. We demonstrate this framework on metalloproteins, where similar coordination chemistry is shared at the structure level yet how sequence controls metal specificity remains underexplored. We develop a three-step workflow: (1) construction of a dictionary of functional motifs; (2) prediction of the next motif and inter-motif gaps; and (3) sequence infilling given the predicted anchor motifs. Motif analysis confirms that the extraction captures known metal-coordination chemistry. Compared to random masking, motif-guided generation improves Conserved Domain Database annotation rates with greater target-family enrichment. Compared to generation approaches that use BPE tokenization, our approach achieves more specific hits on metalloprotein families and substantially reduces off-target annotations. Generation output directly mirrors dictionary composition, demonstrating that motif vocabularies provide explicit control over generation scope. AlphaFold 3 structure prediction with explicit metal ions confirms plausible coordination geometry, validating functional binding sites in generated sequences. Together, functional motif modeling enables interpretable, controllable protein generation, an important step toward compositional design of novel protein functions.