Jul 2026· International journal of computer information systems and industrial management applications· Vol 18, pp. 726-743· 0 citations
TL;DR
This paper proposes a condition number-aware pruning framework that explicitly preserves mathematical stability during the pruning process, and significantly improves both standard accuracy and adversarial robustness compared to conventional pruning methods.
Abstract
The increasing scale of deep neural networks has necessitated model compression techniques, with pruning emerging as a prominent approach to reduce computational and memory costs. However, aggressive pruning introduces a critical challenge: the degradation of mathematical stability and adversarial robustness. Recent research reveals that highly pruned weight matrices tend to become ill-conditioned, exhibiting exploding condition numbers that undermine model performance and robustness . This paper proposes a condition number-aware pruning framework that explicitly preserves mathematical stability during the pruning process. We establish theoretical connections between sparsity, condition number, and local Lipschitz continuity, demonstrating that the condition number becomes the dominant factor limiting robustness in over-sparsified models . Our methodology integrates a differentiable Condition Number Constraint (CNC) with transformed sparse regularization (TSCNC) to simultaneously achieve high sparsity and well-conditioned weight matrices. Experimental evaluations on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that our approach significantly improves both standard accuracy and adversarial robustness compared to conventional pruning methods, achieving superior performance across VGG, ResNet, and WideResNet architectures.
Mutual Heterogeneous Learning (MHL) is proposed, a framework enabling robust pruning via single-model inference that significantly outperforms single-model baselines in both adversarial robustness and corruption robustness, while maintaining competitive clean accuracy.
Jinhui Yu, Zikai Zhang, Khaled A. Harras et al.· 0 citations
Sparse autoencoders (SAEs) are widely used to interpret the internal representations of large language models (LLMs), yet their reliability under post-hoc model compression remains poorly understood. We present a systematic study of how pruning affects SAE behavior and theoretically show that, for a fixed SAE, its impact is governed by perturbation energy, a covariance-weighted norm. This perspective exposes a key limitation of magnitude pruning: by ignoring activation geometry, it distorts the learned representation space and degrades SAE functionality. Activation-aware methods such as Wanda and SparseGPT, in contrast, implicitly control perturbation energy and are therefore substantially more robust at preserving SAE behavior. We further reveal a consistent structural vulnerability across all pruning methods: middle layers are significantly more sensitive to pruning than early or late layers. Guided by this insight, we propose a layer-wise sparsity allocation strategy, achieving lower perplexity under the same average pruning sparsity. Experiments across four model architectures validate our theoretical findings. Code is publicly available at https://github.com/osu-srml/sae-robustness-under-pruning/tree/main.
Suchit Gupte, Xueru Zhang, M. Khalili· 0 citations
Over-parameterization is critical for optimizing neural networks, whereas training sparse networks directly often fails to achieve satisfactory performance. However, the Lottery Ticket Hypothesis (LTH) demonstrates that a randomly initialized dense model has a sparse subnetwork that can be identified through iterative magnitude pruning, while maintaining strong trainability. In this study, we examine a complementary perspective on LTH and propose the Dual Lottery Ticket Hypothesis (DLTH), enabling general training for sparse networks. Specifically, DLTH asserts that any randomly chosen subnetwork drawn from its dense counterparts can be converted to a state with promising trainability. To validate DLTH, we introduce Sharpness-Aware Random Sparse Network Transformation (SA-RST), which uses a regularization term to extrude information from weights slated for masking. Further, this extrusion process is dynamically adjusted based on model sharpness during optimization. Experiments using both CNN and Transformer architectures across several benchmarks demonstrate the effectiveness of our SA-RST and validate the proposed DLTH. Furthermore, we provide comprehensive comparisons with dynamic sparse training methods (RigL, SET) and conduct ablation studies on the sharpness-aware component to validate our design choices. Our study is expected to yield new insights into general sparse network training.
Yue Bai, Mingyuan Zhang, Huan Wang et al.· IEEE Transactions on Pattern...· 0 citations
This paper proposes a novel compression method that uses Singular Value Decomposition (SVD) informed by data examples that achieves substantial reductions in model size and computation with minimal impact on accuracy and demonstrates its applicability in compressing deep neural networks to a fraction of their original size.
Abdelfattah Toulaoui, H. Khalfi, I. Hafidi· Journal of Artificial Intell...· 0 citations
Domain generalization (DG) and neural network pruning are conventionally treated as distinct objectives, targeting out-of-distribution (OOD) robustness and model efficiency, respectively. In this work, we bridge this gap by introducing Domain-Aware Pruning (DAP), a framework that leverages network sparsity as a mechanism to implicitly enhance generalization to unseen domains. Diverging from standard binary mask optimization, DAP learns a continuous parameter retention probability $p \in [0, 1]$, framing network compression as a continuous probabilistic masking problem. By introducing a regularization objective that actively penalizes the retention of domain-sensitive weights during the mask training, DAP identifies a domain-invariant subnetwork. Empirical results across five DG benchmark datasets demonstrate that DAP achieves significant sparsity while consistently matching or exceeding the OOD performance of its dense counterparts. Crucially, DAP is an algorithm-agnostic framework that integrates seamlessly with existing DG pipelines without necessitating post-hoc fine-tuning. Beyond efficiency and generalization, we show that DAP natively provides increased robustness to adversarial perturbations and yields highly interpretable models, where the retained weights reliably encapsulate the most domain-invariant and task-critical representations.
Generative Adversarial Networks (GANs) are widely used for generative modeling in neural networks, yet their practical use is hindered by training instability and mode collapse. Existing stabilization approaches predominantly rely on enforcing fixed 1-Lipschitz constraints on the discriminator through techniques such as gradient penalty, spectral normalization, and gradient normalization. However, these methods fail to account for the dynamic nature of neural network training, where the appropriate Lipschitz constant target may vary with evolving system dynamics. This work challenges the conventional fixed-constraint paradigm by formalizing how the effective constraint strength and a bound-implied Lipschitz constant target can depend on architecture and training state. We provide theoretical analysis to motivate adaptive Lipschitz constraint control and introduce Adaptive Lipschitz Constraint (ALC), a modular mechanism that adjusts the Lipschitz constant target rather than introducing another standalone regularization term. Experiments across multiple stable GAN architectures show improvements in generation quality and training stability in several evaluated settings. Our contributions advance the understanding of neural system stability in adversarial learning and provide a practical approach to adaptive Lipschitz constraint control in adversarial objectives.
Juexin Zhang, Ying Weng, Xueping Zhao et al.· International Journal of Neu...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
