A machine learning framework for enhancing threat detection in core banking systems

Abstract

Abstract One of the biggest issues around using machine learning (ML) for cybersecurity in commercial banking is the gap between proof-of-concept and in production. Not every model that has a high accuracy score on a benchmark dataset is the same model as one that is operating in a live transaction pipeline, subject to the multiple regulatory regimes including the General Data Protection Regulation (GDPR), Payment Card Industry Data Security Standard (PCI-DSS), New York Department of Financial Services (NYDFS) Part 500, Gramm–Leach–Bliley Act (GLBA), NIST Cybersecurity Framework (CSF), Basel Committee on Banking Supervision (BCBS) operational resilience principles and the European Union Artificial Intelligence Act (EU AI Act), which has requirements that go far beyond predictive accuracy. In this paper, a twelve-month experience of designing, deployment, and operation of an ML-based security framework has been reported, that serves more than one million Customers of Midland Bank Plc, Dhaka, Bangladesh on its Core Banking System (CBS) platform. The created detection stack consists of two models, a Convolutional Neural Network (CNN) for spatial anomaly detection and a Long Short-Term Memory (LSTM) network for sequential patterns analysis, and two other methods, namely the Autoencoder (AE) for anomaly scoring via reconstruction errors and the Random Forest (RF) and Gradient Boosting (GB) ensemble for final decision making. Evaluation is based upon dataset of 4,286,530 anonymised events (6 categories of threat instances partitioned in a stratified manner as 70/15/15) with 5-fold cross validation approach. During production, the system had 98.0% ± 0.7% (95% confidence interval [97.4%, 98.6%]) detection accuracy with a false-positive rate of 2%. FPR and a mean response time of 5 min and Area Under the Curve (AUC) of 0.98. There is evidence of a decrease in FPR from 32% to 2%, a decrease in mean response time from 45 min to 5 min and an increase in composite resilience score from 55 to 94, as proven by before and after metrics from Midland Bank Plc. Through ablation analysis, we verify the contributions of the complementary components, and are able to keep competitive accuracy versus 2024–2026 state-of-the-art Graph Neural Network (GNN), transformer and federated-learning approaches while reducing inference latency by 2–3× and estimated deployment cost by 3–7×. From the 1,100 incidents identified during deployment, 30.9% of were phishing/malware incidents, 18.2% DDoS attacks and 16.4% Advanced Persistent Threats (APTs). Finally, consolidated, audit-tested guidance for banks implementing ML-based security under various multi-jurisdictional mandates is provided in the paper.