Deep Learning vs. Traditional Machine Learning for Software Defect Prediction: A Meta-Analysis
TL;DR
The evidence supports a conditional, not universal, DL advantage whose practical value depends on architecture, validation protocol, data distribution, tuning quality, and deployment cost.
Abstract
: Deep learning (DL) is now routine in software defect prediction (SDP), yet how much it improves on traditional machine learning (ML), how stable that improvement is, and what governs it remain contested. We synthesized 45 empirical studies published between 2015 and 2024, comprising 1540 performance estimates, using Hedges’ g of the area underx the receiver operating characteristic curve (AUC) and a random-effects model. The pooled effect was g = 0.61 (95% CI 0.53 – 0.70; p < 0.001), but heterogeneity was substantial ( I ² = 87.7%; τ ² = 0.076) and the 95% prediction interval was 0.07 – 1.16. The lower prediction limit is close to the null and is therefore more decision-relevant than the positive mean alone: a new setting may show little practical gain. Hybrid architectures produced the largest subgroup estimate ( g = 0.90), whereas the gain under cross-project defect prediction was less than half that under within-pro-ject defect prediction (0.31 vs. 0.67). A mild publication-year association explained only about 15% of between-study heterogeneity, and no sample-size association was detected. Publication-bias and leave-one-out diagnostics did not identify a single dominant study, but these diagnostics do not eliminate bias from primary-study design or baseline tuning. The evidence therefore supports a conditional, not universal, DL advantage whose practical value depends on architecture, validation protocol, data distribution, tuning quality, and deployment cost.