Actionable insights from developer behavior: a practical approach to software defect prediction
TL;DR
These findings validate cognitive theory for explainable, actionable, and interpretable safety-critical defect prediction, laying empirical groundwork to evaluate analogous issues in LLM-generated code through the behavioral study of AI.
Abstract
Software defect prediction historically relies on code and history metrics, largely ignoring developer-level indicators. Yet, because defects typically stem from human error, human-factors theory offers complementary, theoretically grounded predictors. This paper introduces a framework for deriving prediction metrics from human-factors theory and instantiates it with two metrics, alertness and memory_decay, operationalizing developer cognitive state during code editing. We evaluate these metrics on twenty-one large-scale open-source projects under two complementary protocols: a discovery-oriented (in-sample) evaluation and a time-aware generalization evaluation on a subset of eighteen projects. Under discovery evaluation, models using cognitive metrics dominate state-of-the-art baselines on every project (PR-AUC 0.64 vs. 0.43), and have surfaced over thirty real, previously-unreported defects across nine critical infrastructure projects, including two assigned CVEs and accepted upstream fixes in systems such as NASA Trick. Under time-aware cross-project evaluation, cognitive metrics encode the strongest cross-project predictive signal in the benchmark: alertness ranks first by mean SHAP importance and memory_decay exhibits the lowest rank volatility, a stability independently confirmed by chance-corrected selection-stability analysis; the configuration combining cognitive and traditional metrics significantly outperforms the traditional baseline alone (Holm-corrected \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p = 0.018$$\end{document}, large effect). Furthermore, cognitive configurations demonstrate a significant, large-effect reverse generalization gap—performing better under cross-project transfer than within-project evaluation—while substituting traditional edit-frequency counts to produce calibration-neutral models. Consequently, our findings validate cognitive theory for explainable, actionable, and interpretable safety-critical defect prediction, laying empirical groundwork to evaluate analogous issues in LLM-generated code through the behavioral study of AI.