TEDlm: domain-centric protein language models with optional structural pre-training

Abstract

Conventional protein language models are pretrained on full-length sequences that interleave multiple domains with linkers and disordered regions, diluting fold-specific signals. Our approach pretrains masked language models on structurally-defined domain segments from The Encyclopedia of Domains. TEDlm learns from domain sequences alone with a standard MLM objective, while its variant TEDlm3D adds a Cα distance-guided contact loss that supervises the attention maps. On CATH S40 remote-homology detection (<40% identity), the domain-centric pretraining has a bigger effect than model scale: at the final layer, a 650M-parameter TEDlm achieves an AUROC1 of 0.28 compared to 0.22 for ESM2 3B, whereas TEDlm3D reaches 0.50, approaching the structure-based search tool Foldseek (0.53) from sequence alone at inference. Attention-map and categorical Jacobian probes show that the contact signal is encoded in the model representations themselves, not only in a trained output head. TEDlm variants also substantially improve zero-shot Molecular Function prediction over ESM2, while matching it on various biophysical property tasks, indicating that signals are largely domain-intrinsic. Together, these results position domain-centric pretraining as a route to compact, structurally informed protein language models.