This work forms the pruning objective as a cardinality-constrained binary quadratic program that explicitly models cross-unit dependencies in the reconstruction error, and develops a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection.
Abstract
Structured pruning is a promising approach for reducing the substantial inference costs of Large Language Models (LLMs) while maintaining hardware efficiency. Many existing methods assess the importance of prunable units (e.g., channels or heads) in isolation, implicitly assuming that pruning errors are additive. This independence assumption is often invalidated by the non-orthogonality of model weights and strong correlations between unit activations, potentially leading to performance degradation. To address this, we propose a Correlation-Aware Structured Pruning method. We formulate the pruning objective as a cardinality-constrained binary quadratic program that explicitly models cross-unit dependencies in the reconstruction error. Since this binary quadratic program is NP-hard and difficult to solve exactly, we develop a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection. Furthermore, we incorporate a gradient-based strategy to achieve adaptive layer-wise sparsity allocation across the entire model. Extensive experiments on mainstream LLMs demonstrate that incorporating correlation information yields competitive accuracy-efficiency trade-offs compared to representative structured pruning baselines.
This work characterize the effect of a residual-stream perturbation through both its activation covariance and the local sensitivity of the model output, and derives a tractable spectral upper bound that reduces subspace selection to an eigendecomposition of a sensitivity-weighted covariance matrix, retaining the effic...
Chayne Thrash, Ke Chen, Soheil Kolouri· arXiv.org· 0 citations
Debias-SparseGPT is introduced, a post-training pruning method incorporating representational debiasing using a second-order term defined over demographically contrasting inputs that consistently reduces pruning-induced bias compared to SparseGPT while preserving model perplexity and zero-shot accuracy.
Irina Proskurina, Guillaume Metzler, Antoine Gourru et al.· 0 citations
This work introduces StructTune1, a PEFT method that uses entire rows or columns as basic units for identifying task-relevant sparse components and develops a sensitivity-based framework to efficiently identify task-relevant weight structures without full fine-tuning.
Yu-Yang Liu, Lai-Man Po, Kwok-Wai Cheung et al.· Transactions of the Associat...· 0 citations
The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding...
Qi-Tong Wang, Xin-Wei Niu, Min Su et al.· 1 citation
REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation, and a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters.
Peng Liu, Hui-Bing Zeng, Yi-Qun Zhang et al.· IEEE Transactions on Emergin...· 0 citations
A larger scale comparison suggests that the current layer-level MoEP design probably does not inherit the same scalability and efficiency advantages as standard FFN-level MoE, because it routes complete Attention-FFN blocks rather than only feed-forward experts.
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.