Cross-Domain Uncertainty Evaluation in Predictive Analytics: Credit Default and Customer Churn

Abstract

This study develops and evaluates a cross-domain, probability-based framework for evaluating uncertainty in predictive analytics. Although predictive models widely support decision-making, many models are evaluated mainly using accuracy-based indicators that do not show whether predicted probabilities are reliable, calibrated, or suitable for decision support. The proposed framework integrates probabilistic classification, Bayesian band-level estimation, calibration diagnostics, bootstrap validation, scoring rules, and probability-based decision bands. Two public benchmark datasets were used: a credit-card default dataset with 30,000 observations as the main financial-risk case and a customer-churn dataset with 7,043 observations as a secondary marketing-analytics validation case. Logistic regression, random forest, and gradient boosting models were compared using conventional metrics and uncertainty-aware indicators, including the area under the receiver operating characteristic curve (ROC-AUC), accuracy, precision, recall, F1-score, Brier score, log loss, expected calibration error, maximum calibration error, bootstrap intervals, calibration curves, and Bayesian credible intervals for observed event rates within probability bands. The results show that gradient boosting provided the strongest probability reliability in both empirical cases, with a Brier score of 0.1380 and expected calibration error of 0.0179 in credit default prediction and a Brier score of 0.1407 and expected calibration error of 0.0346 in customer churn prediction. Decision-band analysis showed that borderline cases had observed event rates near 54.33% for credit default and 52.43% for customer churn, confirming that uncertainty bands can separate confident predictions from review-needed cases. The study advances trustworthy predictive analytics by showing that model evaluation should combine predictive accuracy with probability reliability, calibration quality, and uncertainty communication.