Protein-novelty, sequence-similarity, and negative-sampling controls reveal benchmark-specific discrimination in protein–protein interaction prediction

Abstract

Protein–protein interaction (PPI) predictors are often evaluated with random pair splits that permit proteins to recur across training and test sets, while constructed negative pairs can introduce a separate source of benchmark-specific signal. We reassessed the Guo yeast benchmark using a matched protein-novelty curve, independently generated MMseqs2 sequence-cluster-controlled R3 variants, topology-only controls, degree- and length-matched putative negatives, handcrafted and frozen protein-language-model representations, multiple model configurations, seed analyses, and connected-component-aware inference. In the matched R1→R2→R3 comparison, AAC+CTD–MLP ROC-AUC declined from 0.697 for seen–seen pairs to 0.626 for one-unseen pairs and 0.587 for both-unseen pairs. Across independently regenerated cluster-disjoint R3 variants at 50%, 40%, 30%, and 20% MMseqs2 clustering thresholds, performance showed no consistent monotonic trend. Guo original-negative endpoints had lower positive-network degree than positive endpoints (mean 3.34 versus 6.72), and a degree-only logistic model reached ROC-AUC 0.962 on the original negatives but 0.502 on degree-matched negatives. We used degree-matched negatives consistently for R3 training and testing; six sequence-based model configurations yielded ROC-AUC 0.493–0.547; none showed corrected connected-component-aware evidence of above-chance discrimination. A PIPR multispecies analysis reproduced degree-associated label separation but also exposed cross-species pairing as an additional pooled cue. These results show that protein novelty, operational sequence-cluster separation, and negative-set topology are distinct dimensions of PPI benchmark evaluation. The Guo result is benchmark-specific and should not be generalized to sequence-based PPI prediction or protein language models overall. The dataset and source code for the experiments are publicly available at https://github.com/morufwork/guoAudit.git. Key Points Random pair splits, protein-disjoint splits, sequence-cluster controls, and negative-set controls answer different evaluation questions and should be reported separately. In Guo, AAC+CTD–MLP ROC-AUC declined from 0.697 at R1 to 0.587 at R3, while independently generated R3-50/40/30/20 variants showed no monotonic threshold trend. Guo original negatives carried strong positive-network-degree imbalance; a degree-only logistic model achieved ROC-AUC 0.962 on N0 but 0.502 on degree-matched N2. Under consistent R3/N2 evaluation, six sequence-model configurations had ROC-AUC 0.493–0.547, and the secondary PIPR analysis showed that pooled degree matching can introduce an additional taxonomic-pairing cue.