The Impact of Target Dataset Choice on Machine Learning Prediction of Severe Hail, Wind, and Tornadoes

Abstract

Supervised machine learning (ML) methods implicitly assume that their target dataset is error free. For severe weather prediction, point-based observed Storm Data storm reports that have been expanded in space and time are commonly-used targets. However, other target datasets could potentially better represent severe weather occurrence, especially for situations requiring more precision. This study analyzes 1-hour probabilistic severe weather hazard forecasts made at lead times from 0-3 hours from random forests (RFs). All RFs are trained using predictors from the Warn-on-Forecast System, ProbSevere Version 3, and the Tornado Probability Algorithm. However, some RFs use spatiotemporally expanded reports as targets, while others use hazard-specific National Weather Service warnings. Both target datasets are constructed to have the same overall climatological relative frequencies for a given hazard but have different spatial and temporal climatologies. Results show that RFs tend to produce predictions that retain the climatological characteristics of their target dataset. For example, warnings-trained RFs produced higher (lower) probabilities after (before) 00 UTC compared to report-trained RFs. Qualitatively, forecast probabilities from the warnings-trained RFs tend to be sharper and more spatially focused, especially for severe hail and wind. Additionally, RFs trained and verified using warnings produce forecasts with better objective verification. However, warnings-trained RF forecasts verified on reports have notably worse objective verification than report-trained RF forecasts verified on warnings. Overall, results highlight the importance of target dataset selection during ML model development and evaluation.