An explainable digital twin framework for structural health monitoring: decision-level fusion of vibration and image diagnostics using SHAP and Grad-CAM

Abstract

Structural health monitoring systems based on machine learning routinely achieve high classification accuracy but rarely explain the basis of their decisions, limiting their adoption by practicing engineers who must justify safety-critical actions. This paper presents a framework that combines vibration-based and image-based damage assessment with explainable artificial intelligence and a data-driven digital twin. An XGBoost classifier is trained on physics-informed modal features extracted from 15 months of monitoring data from the KW51 railway bridge in Leuven, Belgium, to distinguish healthy, damaged, and repaired structural states, while Shapley Additive Explanations are used to attribute each prediction to specific frequency, damping, and environmental features. In parallel, an EfficientNet-B0 convolutional neural network is fine-tuned on the Concrete Defect Bridge Image dataset to classify six concrete surface defect types, and Gradient-weighted Class Activation Mapping is used to visually localize the image regions driving each defect prediction. The outputs of the two branches are combined into a Structural Health Index designed to support a continuously updated data-driven digital twin. On a temporally held-out test set, the vibration classifier achieves a macro F1 score of 0.682 and an area under the receiver operating characteristic curve of 0.880, while the image classifier achieves a macro F1 score of 0.886 and an area under the curve of 0.977. The study also identifies three findings with direct relevance for deployment: environmental variables retain a measurable share of feature importance after detrending, the vibration classifier confuses the damaged and repaired classes in a safety-relevant direction, and a fixed-weight fusion of the two branches on unpaired data does not improve upon the vibration branch alone. Together, these findings highlight the importance of paired multi-modal datasets for reliable fusion and provide practical guidance for the development of robust, explainable, and validated digital twin frameworks for structural health monitoring.