Diverge to Converge: Mutual Heterogeneous Learning for Robust Pruning
TL;DR
Mutual Heterogeneous Learning (MHL) is proposed, a framework enabling robust pruning via single-model inference that significantly outperforms single-model baselines in both adversarial robustness and corruption robustness, while maintaining competitive clean accuracy.
Abstract
Neural network pruning is crucial for efficient deployment on resource-constrained devices, yet achieving high sparsity often leads to significant robustness degradation against adversarial perturbations and corruptions. Recent works typically rely on single-model fine-tuning along a fixed optimization trajectory, which renders the network susceptible to local optima and noise while failing to restore the multiple robustness properties compromised during compression. In this paper, we propose Mutual Heterogeneous Learning (MHL), a framework enabling robust pruning via single-model inference. MHL instantiates heterogeneity through two complementary mechanisms: layer-wise Lipschitz regularization for intermediate feature smoothness and adaptive margin objective for difficulty-aware boundary separation. To guide these diverse experts to converge, we employ entropy-based mutual distillation with a strategic schedule that shifts the optimization trajectory from exploring diverse feature subspaces to consolidating a unified robust model. Extensive experiments on 4 clean and corruption benchmarks and adversarial attacks demonstrate that MHL significantly outperforms single-model baselines in both adversarial robustness (+5%) and corruption robustness (+2.6%) while maintaining competitive clean accuracy.