Hidden Decisions in Statistical Software: How Default Settings and Implementation Choices Undermine Reproducibility
Abstract
When two researchers analyze the same data with the same statistical model but different software, they expect identical results. This expectation is often violated. We identify and catalog 37 “hidden decisions”—default settings and undocumented implementation choices—that vary across five major statistical packages (SPSS, R, SAS, Stata, and Python/statsmodels) for five widely used methods: the t-test, logistic regression, Poisson GLM, linear mixed models, and Cox proportional hazards regression. Through controlled stress tests on carefully constructed “boundary datasets” where the reference p-value is near 0.05, and through verified five-software follow-up analyses, we show how hidden decisions can flip statistical significance in boundary cases. The most impactful verified decisions are: (1) scale or dispersion handling in Poisson GLMs, (2) tie-handling method in Cox regression (Breslow vs. Efron), (3) standard-error computation for noncanonical GLMs, (4) finite-sample degrees-of-freedom vs. Wald-z inference in mixed models, and (5) boundary-fit diagnostics in mixed models. For each hidden decision, we trace the discrepancy to source code where available, or to documentation and diagnostic behavior otherwise, quantify the effect size on p-values, standard errors, and significance verdicts, and provide practical recommendations. We argue that the reproducibility crisis in empirical research has an underappreciated computational component that can be addressed through transparency requirements in reporting.
