Representation over Architecture: A Protocol-Controlled Attribution of Frozen ESM-2 Embeddings versus Encoder Design for Secondary Structure Prediction on CB513

Abstract

Protein secondary structure prediction (PSSP) on the CB513 benchmark [30] has been near a plateau for close to a decade, and many recent models add protein language-model (pLM) representations on top of increasingly elaborate encoders. How much of a reported gain comes from the representation, and how much from the encoder, is rarely established, because the two are introduced together and the evaluation protocol is not held fixed. We address this attribution question directly. Holding the Zhou–Troyanskaya protocol constant, with filtered CullPDB (5,926 chains) for training, CB513 (514 chains) for testing, and frozen feature tensors, we run a single 2 × 4 × 5 factorial experiment that crosses two input representations (with and without frozen ESM-2 embeddings) against four encoder variants across five matched seeds (40 trained models), and analyse it with a two-way ANOVA. The full model, which fuses frozen per-residue ESM-2 embeddings with the benchmark one-hot and profile features and passes them through dilated temporal convolutions, a bidirectional LSTM, a pre-norm Transformer and a CRF decoder, reaches 87.35 ± 0.29% Q3, 77.07 ± 0.57% Q8 and 84.63 ± 0.56 SOV. The representation is the dominant factor: adding frozen ESM-2 improves the matched full model by +3.00 Q3 (95% CI [2.77, 3.24]; p = 3.8×10−6), +4.35 Q8 ([3.78, 4.93]; p = 3.0×10−5) and +3.22 SOV ([2.39, 4.06]; p = 4.3×10−4). In the ANOVA the representation main effect explains an order of magnitude more variance than the encoder (about 17–20× the sum of squares on every metric). Correcting an earlier single-seed analysis, the encoder main effect is now small but statistically resolved (Q3 p = 3.9×10−5), and it is carried mainly by the temporal-convolutional block; a representation×encoder interaction (Q3 p = 0.012) shows that the encoder matters more for the weaker profile-only representation than once ESM-2 is present. Adding the frozen ESM-2 embeddings on their own, without the benchmark profile and one-hot features, already surpasses the full profile-based model (85.66 versus 84.35 Q3 at the full encoder), so the representation carries most of the result by itself. Replicating the factorial on a second, independent test set (TS115) under the identical protocol preserves the ordering and widens the ESM-2 advantage over profiles from +3.1 to +9.1 Q3, which indicates the effect is not specific to CB513. A similarity-stratified analysis against UniRef50 shows the ESM-2 gain grows with a chain’s identity to the pretraining corpus and is not statistically resolved in the small low-similarity subset, so part of the benefit may reflect pretraining exposure to homologues, a caveat we state rather than hide. A residue-level analysis shows the gain is concentrated in hard chains, short segments and residues near helix/strand boundaries. Within the tested setting (frozen ESM-2 under the Zhou–Troyanskaya CB513 protocol with one downstream model family), most of the measured improvement is associated with the self-supervised sequence representation, and the encoder contributes a small, resolvable remainder; the distinctive result is not that the representation helps but that the controlled decomposition resolves a representation×encoder interaction and localises the gain to hard, boundary residues, which a single-factor study cannot show.