Version context and control construction in machine learning detection of malicious package updates across npm and PyPI

Abstract

Open-source package registries such as npm and PyPI are increasingly targeted by software supply-chain attacks in which a trusted package is compromised through a later release. This study reconstructs each candidate release together with its immediate predecessor in npm and PyPI and evaluates machine-learning detection under package-disjoint validation. Against never-compromised controls selected within ecosystem and matched on candidate archive file count, the model reaches ROC-AUC = 0.801 ± 0.006 and nested grouped F1 = 0.792 (95% CI 0.730–0.845). Performance falls to ROC-AUC = 0.551 when the controls are ordinary updates of the same compromised packages. That gap shows that the evaluated features separate compromised packages from clean packages far better than they separate a malicious update from another update of the same package. A strict temporal hold-out returns F1 = 0.310, and training on one registry and testing on the other gives ROC-AUC = 0.498 from npm to PyPI and 0.630 from PyPI to npm. Version context contributes a small incremental signal, but control construction largely determines apparent performance. The approach is therefore presented as a first-stage screening filter, and the results argue for stronger within-package and temporal evaluation.