Fit one model, report one number: how to analyse a training or intervention study

Abstract

A typical training or intervention study allocates participants to protocols, measures them repeatedly over time, and asks whether one protocol changes the outcome more than another. A typical analysis, such as an analysis of variance or a mixed model, is powerful, but that power is for taking a measurement apart, not for delivering the verdict. Such a model splits each measurement into separate terms: a benchmark, covariate adjustments, a time course, and a term that lets each protocol follow its own time course. The statistical software prints a separate test for every one of these terms, and a common workflow reports those tests as the study’s result. However, the study was built to answer only a single question, usually whether one protocol improves the outcome more than another as training goes on. Our position follows directly: choose that one comparison in advance and test it directly, reading it off the fitted model without first requiring an overall, or omnibus, test to pass. This approach can recover effects the usual workflow can miss, because requiring an overall test to pass first costs a substantial amount of power, as our worked examples show. An overall test remains the right test only when the overall question is the one the study asked. Fit the model as richly as the design demands, then report the one number it was built to produce.