Systematic Hyperparameter Optimization of Machine Learning Classifiers for Imbalanced Classification: An Empirical Study With Grid Search Cross-Validation

Abstract

Optimization of hyperparameters is still an important but relatively ignored stage of machine learning pipelines, and imbalanced classification brings forth three key problems that inspire our investigation into the subject: 1) hyperparameter settings used by default in a library are not tailored for imbalanced labeling and, thus, do not identify the minority class well enough; 2) fine-tuning of classifiers is computationally expensive and highly varying; and 3) single metric-based selection criterion can camouflage low minority class identification. In this paper, a thorough experimental evaluation of grid search cross-validation method is provided in regards to four classifiers: Logistic Regression, Decision Tree, Random Forest, and XGBoost. Each classifier is tuned with a distinct search space (24 to 2,520 hyperparameter combinations) using 5-fold stratified cross-validation optimized for recall. The Telco Customer Churn dataset is used as the main example along with three additional telecoms and banks datasets as a means of transferring the knowledge about the benefits and limitations of hyperparameter optimization into other areas. Changes in performance, stability, computational cost and generalization ability of models before and after optimization are analyzed. It is found out that there exists a certain hierarchy of classifier sensitivity to hyperparameters: ensembles are the most sensitive to their tuning, Logistic Regression is not affected at all. Random Forest and XGBoost yield the greatest increase in recall - 84.4% and 87.0% correspondingly, and significant improvements in ROC-AUC; Logistic Regression, on the contrary, changes its metrics very little. Differences in sensitivities are statistically significant for the ensemble models ( $p\lt 0.001$ , two-proportion $z$ test) but not for Logistic Regression and Decision Tree models. While XGBoost provides the highest performance, Random Forest offers better balance between optimization cost and resulting metrics. Cross-dataset experiment shows that the information about tuned hyperparameters can be successfully transferred to other problems, but with slight performance losses.