Model-Dependent Calibration Effects of Imbalance Handling in Credit Card Fraud Detection

Abstract

Class imbalance correction methods such as class weighting and synthetic oversampling are widely used to improve fraud detection, but they can distort the probability estimates that risk scoring depends on. This study evaluates the calibration cost of two common imbalance handling strategies, inverse frequency class weighting and SMOTE-NC, applied to three model families: Logistic Regression, Random Forest, and XGBoost. Using the Sparkov credit card transaction dataset, comprising 1,296,675 training transactions and 555,719 future test transactions, the analysis follows a chronological four way split that separates model training, calibration fitting, threshold selection, and future testing, so that no evaluation period leaks information into model fitting or calibration. Across the three models, mean raw expected calibration error (ECE) rose from 0.0012 without imbalance handling to 0.0715 with class weighting and 0.0742 with SMOTE-NC, driven mainly by Logistic Regression. Mean raw F1 score declined from 0.680 to 0.648 and 0.622, respectively, showing that neither strategy consistently improved detection performance. The effects were model dependent: class weighting improved the F1 score of XGBoost from 0.7878 to 0.8106 while substantially increasing its calibration error, whereas Random Forest with class weighting retained near ideal calibration. For the five configurations whose raw ECE exceeded 0.01, post hoc isotonic calibration, fitted on a dedicated chronological period, reduced ECE by 96.6% to 99.4% while keeping future test F1 score differences below 0.008 across all nine raw versus calibrated pairs. The calibration gains were robust across multiple ECE binning schemes, including 10, 20, and 50 equal width bins and adaptive equal frequency binning. These results indicate that imbalance handling should be evaluated separately for each model family rather than assumed to help uniformly, and that post hoc calibration should be applied conditionally, only when diagnostics indicate substantial probability distortion, rather than as a default step in every fraud detection pipeline.