Weight Clustering Reveals the Importance of Relative Rank in Large Language Models

Abstract

Large language models (LLMs) contain billions of parameters, but many exact weight values are not essential. We apply scalar $K$-means clustering to pretrained models, replacing each weight matrix with only 16--64 shared values. For Llama~3.1-8B-Instruct and SmolLM2-135M, this preserves strong accuracy without retraining, providing a simple way to compress LLMs on disk. Fine-tuning only the cluster centroids further recovers 30--40\% of the remaining accuracy loss. Using clustering as an analysis tool, we then keep the cluster assignments fixed and perturb only the shared values, to study which of their properties matter most. We find that preserving their \emph{relative rank} (which connections are stronger or weaker than others) is more important than preserving their exact magnitudes. Breaking this ordering greatly increases perplexity, while changing the values but keeping the ordering causes little loss in many middle and late layers. However, rank preservation alone is not sufficient. Model performance also depends on maintaining the variance of the reconstructed weights, while early layers remain more sensitive; perturbing the cluster assignments themselves shows that their spatial geometry is also load-bearing. We observe the same behavior across four LLMs from two architecture families. These results suggest that the ordering of weights, together with their overall variance, plays a central role in LLM compression and robustness.