The Undetected Damage of Quantization on Retrieval and How to Fix It
We show that a quantized model that keeps its classification accuracy still changes $14$ to $46\%$ of its top-1 retrieval results, and that aggregate ranking metrics reveal only part of this damage. We tie this failure to the gap between the two highest model scores and use that gap to decide when to trust a quantized...
Luca Zhou, Alessandro Zirilli, Daniele Solombrino et al.
· 0 citations