The Effects of Low-Bit Quantization on Confidence Calibration in Small Language Models

Abstract

This study looks at how confidence patterns shift when cutting data precision to 4 bits using NF4, applied post-training on Phi-3.5-Mini-Instruct, a small-scale language model packing 3.8 billion parameters. Shrinking precision cuts down memory demands while speeding up output creation; however, what remains unclear is how such squeezing affects the way models rate their own sureness, measured via average prediction strength for each produced word unit. Rather than measuring correctness, the spotlight falls strictly on differences in how certain the system sounds across full-detail versus reduced-bit forms. Evaluation runs on ninety thoughtfully picked prompts split evenly among three kinds: factual assertions, fictional statements, and subjective stances. Hesitation creeps into quantized outputs across the board, their probability scores dipping five point seven six percent on average. Not every category bends the same way under pressure; made-up stories barely shift at all, while claims about reality sag a bit more. Opinions? Those take the hardest hit, certainty plummeting close to ten points lower. The same rules applied throughout, yet outcomes were split wide open based on what kind of knowledge was asked for. Oddly enough, false confidence shows up almost exclusively when facts are invented outright: the slimmed-down model gets things wrong but acts sure of itself, unlike its full-sized counterpart. This odd behavior earns the name “confident hallucination.” You might think shrinking models would show clear drops in accuracy or fluency, but standard metrics miss it completely. What matters instead is whether each guess lines up with the truth that matches how safely such systems can be used. Nowhere near steady, these dips shift with each kind of job. Shrinking a model can quietly erode reliability—something standard tests often miss. Where does it really show up? High-stakes areas rely on consistent results.