#artificial intelligence
May 2026
Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
This work presents a systematic study of scale vectors in LLMs from the perspectives of expressivity, optimization, and architectural structure, and proposes three lightweight and complementary improvements to scale vectors: branch-specific heterogeneity, improved placement around linear mappings, and magnitude-direction reparameterization.
Mingze Wang, Shuchen Zhu, Yuxin Fang et al.
· arXiv.org · 3 citations