Stop When It Matters: Detectability-Guided Microbenchmarking for Performance Regression Testing

Abstract

Performance microbenchmarks are widely used for detecting performance regressions in continuous integration, yet existing execution policies remain computationally expensive. We present detectability-guided microbenchmarking, a principled framework that reformulates benchmark execution as a Minimum-Detectable-Effect (MDE) problem, achieving substantial speedups with formal statistical guarantees. Our approach introduces a two-threshold specification: a lower threshold, δ0, for negligible slowdowns, and an upper threshold, δ1, for meaningful regressions. We theoretically and empirically show that this specification enables bounded false-positive rates below δ0 and guaranteed statistical power around δ1, while dramatically reducing execution time. To realize this framework, we propose a two-phase procedure: Phase 1 adaptively controls execution via online sample-size re-estimation guided by minimum detectable effect, enabling early stop when sufficient statistical resolution is reached and futile abort when target detectability is unattainable within the execution budget; Phase 2 performs regression analysis under the same specification, yielding decisions with provable error bounds. Experiments on 1702 microbenchmarks from three large-scale open-source projects demonstrate up to 93% reduction in execution time while empirical false-positive rates and power closely match their theoretical targets. Our findings reveal that detectability-guided execution offers a theoretically grounded yet practical path toward efficient microbenchmark-based regression detection.