A multi-metric evaluation framework for machine learning credit scoring models aligned with Basel III and IFRS 9 regulatory validation requirements

Abstract

Credit default risk models in banking are predominantly evaluated using discrimination metrics such as AUC and accuracy, which are systematically disconnected from the regulatory performance requirements established under Basel III and IFRS 9. This disconnect creates a gap between academic model evaluation practice and supervisory validation expectations. We address this gap by presenting and empirically demonstrating a multi-metric evaluation framework that extends beyond discrimination to incorporate calibration, stability, stress robustness, and expected credit loss analysis. Five machine learning models, including Logistic Regression with Weight of Evidence scorecard, Random Forest, XGBoost, LightGBM, and a Multi-Layer Perceptron, are evaluated using the UCI Credit Card Default dataset (22.1% default rate) against a six-metric regulatory validation suite comprising AUC, Gini, Kolmogorov-Smirnov statistic, Hosmer-Lemeshow goodness-of-fit, Brier Score, and Population Stability Index. Robustness is assessed through three complementary layers: deterministic ICAAP stress testing across five ordered scenarios, Monte Carlo simulation of 2000 correlated loss paths, and a stylized IFRS 9 Expected Credit Loss staging proxy. While candidate models generally satisfy baseline criteria, evaluation against multi-metric thresholds reveals critical individual deficiencies (e.g., MLP fails the discrimination benchmark with AUC = 0.6964 < 0.70, and XGBoost fails calibration goodness-of-fit with HL p = 0.0146 < 0.05). The WoE LR champion satisfies all selected validation thresholds, achieving AUC = 0.7392, Gini = 0.4785, and PSI = 0.0005. Stress testing modifies the baseline model ranking, with the highest-static-AUC model (RF) suffering a 15.4% AUC collapse under scenario-driven severe stress versus only 5.3% for LR. These findings illustrate that credit risk model evaluation under regulatory frameworks requires multi-metric assessment under both normal and adversarial conditions, and that model selection based solely on discrimination metrics may overlook critical robustness and governance considerations.