Skip to content

Forward-Free LLM Depth Pruning via Weight Redundancy

Sep 2026 · 0 citations · 21 references
Computer Science

TL;DR

Weight-Redundancy Pruning (WRP), a forward-free depth-pruning method that estimates inter-layer redundancy from checkpoint weights to select blocks without calibration data or model forward passes, consistently outperforms existing forward-free magnitude pruning and approaches the performance of activation-based methods.

Abstract

Depth pruning reduces large language model (LLM) inference cost by removing complete Transformer blocks. Activation-based methods collect hidden states through forward passes on calibration data, while existing forward-free methods score each Transformer block separately without measuring similarity between blocks. We propose Weight-Redundancy Pruning (WRP), a forward-free depth-pruning method that estimates inter-layer redundancy from checkpoint weights to select blocks without calibration data or model forward passes. WRP compares attention output and MLP down-projection weights across layers and combines their pairwise similarities with relative projection-scale information. The resulting all-pairs similarity matrix guides layer grouping and block selection. Across multiple pruning settings, model families, and downstream tasks, WRP consistently outperforms existing forward-free magnitude pruning and approaches the performance of activation-based methods.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose \textit{LayerTrim}, a training-free structured pruning method that identifies redundant...

Woo-Sang Lim, Haneul Yoo, Vincent-Daniel Yun et al. · 1 citation
Preprint Oct 2026

SliMOO: Interpretable Multi-Objective Evolutionary Search for LLM Depth Pruning

As large language models (LLMs) grow more powerful, their scale makes them costly to serve. Depth pruning offers a practical route to efficiency by directly reducing memory and latency without hardware-specific support. However, we observe that transformer layers exhibit markedly different task-dependent layer contribu...

Guan-Chen Li, Yi-Xing Xu, Xuan-Wu Yin et al. · 0 citations
#machine learning Preprint Oct 2026

Heterogeneous Structured Pruning Under Resource Constraints for Hardware-Efficient Large Language Models

Training-free pruning removes weights from pretrained large language models to reduce their deployment cost. However, existing approaches typically consider only the total number of retained weights. Consequently, their pruning decisions do not account for how target hardware stores and processes those weights. These m...

C. Barker, A. Paraskeva, J. V. van Rijn et al. · 0 citations
#machine learning Preprint Oct 2026

Adaptive-Depth Is a Finishing Stage

Looped transformers reuse the same layer multiple times, allowing inference compute to be adjusted without changing the model. Recent adaptive-depth methods train the model across multiple loop depths from the beginning of training. We introduce fixed-first training, which varies model depth only during a short final p...

Andrei Piterbarg, Sean McLeish, Dhruv B. Pai et al. · 0 citations

Output-aware Residual Stream Pruning for Large Language Models

This work characterize the effect of a residual-stream perturbation through both its activation covariance and the local sensitivity of the model output, and derives a tractable spectral upper bound that reduces subspace selection to an eigendecomposition of a sensitivity-weighted covariance matrix, retaining the effic...

Chayne Thrash, Ke Chen, Soheil Kolouri · 0 citations

GSM: Efficient Language Modeling with Shared Global State

The Global State Model (GSM), a causal encoder--decoder architecture that concentrates the selection and aggregation of long-range information in the encoding stage, is introduced, offering a shared-state architecture for efficient language modeling.

Yun Zheng, Bin Wen, Xiao-Jie Wang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.