ShrinkLLM: Automated LLM Compression via AI Agents

Abstract

Deploying large language models in resource-constrained environments such as mobile devices requires effective compression. Pruning and quantization are largely used, yet applying pruning to a new model typically demands manual inspection of its architecture to identify layer names, MLP structure, and projection patterns, a process that is tedious, error-prone, and must be repeated for every new model design. We present ShrinkLLM, an end-to-end compression system that eliminates the manual inspection of model structure during pruning by delegating architectural analysis to an AI agent. Given a causal LLM, the agent automatically identifies MLP layer organization, expansion layers, gate projections for Gated Linear Unit architectures, and down-projection layers, structural information that is then passed directly to a compression engine to perform neuron pruning, quantization, or a combination of both. We demonstrate the system on llama-3.1-8b-instruct, Qwen2.5-3b-instruct and Gemma-2-2b-it showcasing how the agent correctly resolves architecturespecific structural details and drives compression across all MLP layers automatically.