GroupMask: Layer-Adaptive Group-wise Sparsity for Semi-Structured LLM Pruning
GroupMask is proposed, which generates the group selectors of all layers with a lightweight hypernetwork, relaxes them with a Gumbel-Sigmoid parameterization and a straight-through estimator, and learns them through sparsity-budget regularization and self-distillation while keeping the pretrained weights frozen.
Zhen-Gao Li, Shuo-Qiu Li, Xiao-Fan Zhang et al.
· 0 citations