Skip to content

Correlation-Aware Structured Pruning for Large Language Models

Aug 2026 · 0 citations · 24 references
Computer Science

TL;DR

This work forms the pruning objective as a cardinality-constrained binary quadratic program that explicitly models cross-unit dependencies in the reconstruction error, and develops a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection.

Abstract

Structured pruning is a promising approach for reducing the substantial inference costs of Large Language Models (LLMs) while maintaining hardware efficiency. Many existing methods assess the importance of prunable units (e.g., channels or heads) in isolation, implicitly assuming that pruning errors are additive. This independence assumption is often invalidated by the non-orthogonality of model weights and strong correlations between unit activations, potentially leading to performance degradation. To address this, we propose a Correlation-Aware Structured Pruning method. We formulate the pruning objective as a cardinality-constrained binary quadratic program that explicitly models cross-unit dependencies in the reconstruction error. Since this binary quadratic program is NP-hard and difficult to solve exactly, we develop a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection. Furthermore, we incorporate a gradient-based strategy to achieve adaptive layer-wise sparsity allocation across the entire model. Extensive experiments on mainstream LLMs demonstrate that incorporating correlation information yields competitive accuracy-efficiency trade-offs compared to representative structured pruning baselines.

View source

Similar papers

Output-aware Residual Stream Pruning for Large Language Models

This work characterize the effect of a residual-stream perturbation through both its activation covariance and the local sensitivity of the model output, and derives a tractable spectral upper bound that reduces subspace selection to an eigendecomposition of a sensitivity-weighted covariance matrix, retaining the effic...

Chayne Thrash, Ke Chen, Soheil Kolouri · 0 citations
#natural language process... Preprint Sep 2026

Debias-SparseGPT: Bias-Aware Pruning for Large Language Models

Debias-SparseGPT is introduced, a post-training pruning method incorporating representational debiasing using a second-order term defined over demographically contrasting inputs that consistently reduces pruning-induced bias compared to SparseGPT while preserving model perplexity and zero-shot accuracy.

Irina Proskurina, Guillaume Metzler, Antoine Gourru et al. · 0 citations
Open access 2026

StructTune: Structured Sparse Fine-Tuning for Efficient Adaptation of Large Language Models

This work introduces StructTune1, a PEFT method that uses entire rows or columns as basic units for identifying task-relevant sparse components and develops a sensitivity-based framework to efficiently identify task-relevant weight structures without full fine-tuning.

Yu-Yang Liu, Lai-Man Po, Kwok-Wai Cheung et al. · 0 citations
#machine learning Preprint Oct 2026

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding...

Qi-Tong Wang, Xin-Wei Niu, Min Su et al. · 1 citation
Open access Aug 2026

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation, and a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters.

Peng Liu, Hui-Bing Zeng, Yi-Qun Zhang et al. · 0 citations
Open access Sep 2026

MoEP: Compact and efficient sparsity with modular expert paths.

A larger scale comparison suggests that the current layer-level MoEP design probably does not inherit the same scalability and efficiency advantages as standard FFN-level MoE, because it routes complete Attention-FFN blocks rather than only feed-forward experts.

Joonas Tapaninaho, Mourad Oussalah · 0 citations

Related blog posts

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.