Auxiliary-table timing, not splitter choice, dominates leakage in donor propensity models: evidence from synthetic and real fundraising panels

Abstract

Donor-propensity models built from CRM exports routinely join in auxiliary tables — clinical encounters, event attendance, wealth-screening appends — that arrive as a separate table passed to a transformer's constructor rather than as rows of X. Because those rows are never part of any train/test split, standard leakage defenses (Pipeline, cross_val_score, GroupKFold) have no jurisdiction over them at all: a transformer can aggregate the whole auxiliary table, including rows that postdate the gift being scored, and no splitter will ever see it happen. We measure how much this costs against the standard prescription of splitter choice, on both a seeded synthetic donor-year panel (3,000 donors, six years, five seeds) and a real donor file (KDD Cup 1998, 95,412 donors, reshaped into a 22-period mail/response panel). Building auxiliary features over the whole export rather than as of each decision point inflates walk-forward ROC-AUC by +0.126 AUC on synthetic data and +0.376 AUC on real data (0.482 to 0.858) — three times larger, not smaller as we predicted before running the real-data experiment. Splitter choice, the leakage source practitioner folklore usually names, costs an order of magnitude less on both panels. We report the software (PhilanthroPy, an open-source scikit-learn–native toolkit) that operationalizes the fix as a single as_of cutoff parameter, and the disagreement between our prediction and the result, rather than revising the prediction after the fact.