Models in the War Room: Translating random forest predictions into player comps

Abstract

Interpretability is important for the adoption of complex predictive models, particularly in the “war rooms” where sports executives discuss high-stakes decisions while balancing model predictions with qualitative scouting. Especially in the context of player entry drafts, scouts often explain their assessments of prospects by identifying historical comparable prospects, or “comps”. Lin and Jeon (2006) showed random forests are interpretable as adaptive nearest neighbors and that their predictions are recoverable as weighted averages of training outcomes. In this paper, we draw a line from this decomposition of random forest predictions to the concept of player comps, and we extract similarity scores quantifying the weight of each comp in the prediction for each prospect. This case-based approach to interpretability translates random forest predictions into the language of traditional scouting, presenting an alternative to popular feature-based approaches like SHAP values. We illustrate this approach with an application to a two-stage random forest model for predicting the professional career outcomes of collegiate quarterback prospects in American football. We provide the R package treecomp to help researchers and practitioners to extract similarity scores from their own random forest models.