Evaluating DL Test Adequacy Metrics: A Correlation Study with DeepCrime’s Mutation Score
Evaluating the quality of test suites for Deep Learning (DL) systems generally relies either on structural coverage criteria (e.g., Neuron Coverage), distribution-aware metrics (e.g., Surprise Adequacy), or mutation testing. While mutation testing offers the most reliable measure of fault-detection, it is often prohibitively expensive, as each mutant configuration may require retraining the model from scratch multiple times to establish detectability. This paper investigates to what degree the computationally cheaper coverage and distribution metrics can serve as reliable proxies for mutation testing. We measured the correlation (Spearman and Pearson) between nine adequacy metrics and the DeepCrime mutation score across four datasets (two classification, two regression) at ten test set size proportions. For classification tasks, all metrics except NeuraL Coverage (NLC) demonstrate moderate-to-strong, statistically significant correlations (Spearman 0.53 to 0.69) with the mutation score. For regression tasks, most metrics yield moderate, statistically significant positive Spearman correlations with growing test sets, though partial correlation analysis suggests these are largely mediated by test set size. Our findings suggest that standard adequacy metrics can nevertheless serve as proxies for mutation testing, particularly in classification DL systems.