Skip to content

Locality-Aware Redundancy Pruning for LLM Depth Compression

May 2026 · arXiv.org · Vol abs/2605.27786 · 1 citation · ⚡ 1 influential · 37 references
Computer Science

TL;DR

It is shown that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture, and Representation Locality Score (RLS) is introduced, derived from global inter-layer hidden-state similarity.

Abstract

Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing one-shot pruning methods rely on local layer importance or fixed redundancy assumptions across architectures. We propose Locality-Aware Redundancy Pruning (LoRP), a training-free one-shot depth pruning framework guided by representation locality. We show that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture. To characterize this phenomenon, we introduce Representation Locality Score (RLS), derived from global inter-layer hidden-state similarity. Using a small calibration set, LoRP computes pairwise layer similarity, clusters layers by representational similarity, and allocates pruning according to residual intra-cluster redundancy. Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy. Official github repository: https://github.com/daniel-eai/LoRP-Locality-Aware-Redundancy-Pruning/

View source

Similar papers

#machine learning Preprint Aug 2026

Correlation-Aware Structured Pruning for Large Language Models

This work forms the pruning objective as a cardinality-constrained binary quadratic program that explicitly models cross-unit dependencies in the reconstruction error, and develops a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection.

Si-Cheng Xu, Hao Shi, Wei Zhang et al. · 0 citations
Sep 2026

P4Q: Co-designing Token Pruning and Quantization for Vision-Language Model Acceleration

Vision language models have achieved strong performance across a wide range of multimodal applications, yet their substantial computational and memory costs hinder efficient deployment. Visual token pruning and post-training quantization reduce inference overhead along two complementary dimensions, namely sequence leng...

Hai-Zhao Jing, Zhen-Hao Shang, Hao-Kui Zhang et al. · 0 citations
#small language model Preprint Aug 2026

SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs

SHIFT-LLM, a training-free post-pruning correction framework that inserts a Linear Residual Adapter at each pruning site, consistently recovers accuracy lost to depth pruning across most configurations, achieving gains up to +15.7 points on Llama-3.1-8B-Instruct.

Ali Bahri, Hang Li, Hongliang Li et al. · 0 citations
Open access 2026

Revisiting Structural Pruning via Layer-Wise Feature Map Similarity for Edge AI: A Self-Adaptive Methodology to Minimize Search Overhead

Efficient deployment of Deep Neural Networks (DNNs) on edge devices requires effective model compression with low optimization cost. However, existing approaches often force users into a costly trial-and-error process to find a suitable balance between compression and accuracy. This paper presents Self-Adaptive Structu...

Milad Kokhazadeh, Vasilios I. Kelefouras, Panagiotis G. Mousouliotis et al. · 0 citations
Conference Open access Sep 2026

From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankers

RaDiCal (Rank-Discriminative Calibration), a training-free framework that uses normalized attention entropy to decide when saliency can be trusted, fusing it with an attention-free rank-discriminative prior and selecting pruning layers from the same trust landscape is proposed.

Si-Yi Liu, Han-Jun Yang, Chen-Chen Zhang et al. · 1 citation
#artificial intelligence Preprint Sep 2026

Forward-Free LLM Depth Pruning via Weight Redundancy

Weight-Redundancy Pruning (WRP), a forward-free depth-pruning method that estimates inter-layer redundancy from checkpoint weights to select blocks without calibration data or model forward passes, consistently outperforms existing forward-free magnitude pruning and approaches the performance of activation-based method...

Vincent-Daniel Yun, Woo-Sang Lim · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.