Explainable AI for malware opcode sequence analysis and explainability-motivated saliency-map based spurious correlation robustness in images

Abstract

Explainability offers a powerful lens for understanding and improving the robustness of Machine Learning (ML) models. This work demonstrates how eXplainable AI (XAI) techniques can be used not only to interpret model behaviour, but also to develop robust training algorithms that encourage the learning of semantically meaningful features. The first contribution is Hierarchical-LIME (H-LIME), a novel XAI method tailored to malicious Android opcode sequence analysis. Unlike standard LIME, which treats individual opcodes as flat, independent features, H-LIME leverages the hierarchical structure of programs, such as classes and methods, to produce sparser and more descriptively accurate explanations, improving the localisation of malicious code segments. The second contribution, UnLearning from Experience (ULE), integrates explainability into the training process of image classifiers by leveraging saliency maps to guide model behaviour. ULE trains two models concurrently: a student model and a teacher model. The student is trained using standard Empirical Risk Minimisation and learns to rely on spurious correlations present in the data. Meanwhile, the teacher is trained to actively avoid these spurious correlations by minimising alignment with the student’s saliency maps. This encourages the teacher to focus on more semantically meaningful features, resulting in a model that learns a more robust feature representation. Importantly, ULE achieves this without requiring x access to group labels or prior information about spurious features, making it widely applicable in real-world scenarios. The final contribution, Weighted UnLearning from Experience (wULE), builds upon ULE by introducing a targeted sample re-weighting strategy that distinguishes between samples likely and unlikely to contain spurious correlations. Instead of treating all samples uniformly, wULE leverages the simplicity bias principle to estimate the set of training samples suspected to contain spurious correlations. The loss function is adjusted on a per-sample basis: samples in this set receive stronger penalties for gradient alignment to encourage unlearning, while cleaner samples are weighted more heavily in the classification objective to reinforce reliable feature learning. This adaptive strategy leads to improved worst-group performance and further enhances interpretability through more focused and meaningful saliency maps. Together, these contributions position explainability, not just as a diagnostic tool, but a core strategy for training resilient and robust machine learning systems.