DecayBench: A Reference-Free Benchmark for Trustworthy Drift Detection

Abstract

Distribution drift can substantially degrade the performance of deployed machine learning models; for example, accuracy on SST-2 can fall from 88% to 58%. Detecting such degradation is fundamentally challenging because deployment provides inputs but not labels, so the detection itself must be reference-free. We introduce DecayBench, the first reference-free, calibrated benchmark for evaluating drift detectors. DecayBench measures detector trustworthiness along five axes (calibrated, valid, timely, no-regret, adaptive), and compares ten existing detectors across ten NLP, vision, and multimodal datasets using paired-bootstrap significance testing. Evaluation on DecayBench shows that no existing detector is uniformly optimal. Motivated by this observation, we propose Alert, a label-free aggregation rule for drift detection. Unlike all competing combiners, it uses a label-free self-configuring selection rule with a no-regret guarantee. Alert has three contributions: (i) a dilution analysis yielding a self-configuring detector selection rule; (ii) a finite-sample conformal guarantee that controls the false-alarm probability on clean data at any prescribed level (e.g., 5%) for arbitrary score distributions; and (iii) a no-regret result: when no single detector dominates (constituents of comparable effect size, a condition checkable offline), Alert matches or beats the best constituent, being never significantly worse and sometimes better by a large margin; this holds across NLP, NLI, and vision (ResNet), with the largest gains under multimodal drift, and the proof identifies a dominant single detector (MMD on CLIP) as the only dilution exception. We prove the no-regret property and, across the benchmark, report its empirical counterpart, non-dominance under a paired bootstrap (Alert is never significantly worse than the best constituent), which at some operating points is statistically inconclusive rather than a strict win. Because Alert combines only embedding- and logit-based detector scores, it directly transfers across NLP, vision, and multimodal models. Empirically, Alert strictly improves over single-modality monitoring, increasing AUC by up to 25 points under mixed-modality drift and by approximately 50 points under cross-modal mismatch, where individual modality-specific detectors perform near chance. Alert also matches or outperforms the Fisher, Simes, Bonferroni, and median combiners, performs best under low-severity drift, and matches or surpasses early fusion (Concat-MMD) in both multimodal settings.