SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability
Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quantization often suffers severe accuracy loss, while vector quantization (VQ) better preserves accuracy through more expressive joint encoding. However, existing VQ methods...
Xiang-Long Yan, Ze-Nan Wu, Ke-Wei Zhang et al.
· 0 citations