Big Data- and AI-Driven Hybrid Self-Attention Credit Scoring with Explainable Decisioning
TL;DR
This work proposes a tenure-stratified hybrid framework that couples an online weight-of-evidence logistic regression (WOE-LR) scorecard with an offline self-attention stacked ensemble whose calibrated PD is quantile-binned, WOE-encoded, and re-injected into the online scorecard as a single auditable predictor.
Abstract
Real-time retail credit scoring is a data-intensive cognitive computing task. Each decision must fuse heterogeneous signals, execute a non-linear model, return a calibrated probability of default (PD), and emit a regulator-compliant local explanation within milliseconds. We address the most demanding segment of unsecured lending in Kazakhstan—Salary-Project-Independent (SPI) borrowers, whose principal income stream is not observable by the lender—and frame scoring as a constrained optimisation problem where we maximise discrimination subject to interpretability, latency, and calibration constraints. We propose a tenure-stratified hybrid framework that couples (i) an online weight-of-evidence logistic regression (WOE-LR) scorecard with (ii) an offline self-attention stacked ensemble (LightGBM, CatBoost, and a tabular self-attention network) whose calibrated PD is quantile-binned, WOE-encoded, and re-injected into the online scorecard as a single auditable predictor. On 551,962 production contracts that originated in 2022–2024, the repeat-client hybrid attains an area under the receiver operating characteristic curve (AUROC) of 0.826, a Gini coefficient of 0.65, and a Kolmogorov–Smirnov (KS) statistic of 0.495, preserving roughly half of the offline ensemble’s lift over the linear baseline (AUROC 0.79→0.897) while retaining a fully auditable twelve-coefficient scorecard in production. The new-client scorecard attains an AUROC of 0.741. Non-parametric isotonic recalibration reduces the expected calibration error from 0.27 to below 0.01 and raises the Hosmer–Lemeshow p-value above 0.99 without altering discrimination. The framework complies with the model risk standards of the Agency of the Republic of Kazakhstan for Regulation and Development of the Financial Market and is delivered as a Spark/MLOps reference architecture, illustrating how big data engineering, attention-based representation learning, and post hoc explanations can be co-designed for a high-stakes, high-throughput, regulated AI application.