Sub-Two-Bit Ternary Weight Storage with Table-Free Arithmetic Decoding for CPU Inference

Abstract

Ternary (1.58-bit) language models are stored in practice at two bits per weight, 26% above the information content of a ternary symbol. The code that removes that redundancy is not new: five ternary values as the base-3 digits of one byte at 1.600 bits per weight is llama.cpp's TQ1_0 and Spectra 1.1's TQ1; the depth-one, table-free decode is used by the NTRU reference implementation; the condition under which a denser format pays is due to Zukowski et al. What this paper offers is measurement. We build t5b, the same construction at a block length of 160 with the scale carried in the existing i2_s per-tensor tail, integrate it into llama.cpp as a distinct ggml type alongside the shipped two-bit format, and measure it end to end on BitNet-b1.58-2B-4T. The re-encoding is lossless: output is bit-identical, and perplexity over 20,480 tokens of WikiText-2 agrees with the two-bit baseline to six significant figures. The checkpoint shrinks from 1.10 GiB to 1.00 GiB, and under llama-bench at four threads the median over five invocations is 1.119x prompt and 1.133x generation throughput, faster in five of five. Against TQ1_0 on the same machine the size margin is 4.7% of the ternary payload. The profitability condition is measured rather than assumed, with both terms taken from the part the kernel runs on; a depth-one decode transplanted into upstream's TQ1_0 kernel does not pay there. We further report that 4.76% of the model's feed-forward neurons are identically zero, in coinciding gate/up index sets in all thirty layers, surviving an independent fine-tune exactly. All results are from one AVX2 host without VNNI; a reviewer's runs on an Intel Xeon reproduce the arithmetic ratio and show it rising by 8.8% under VNNI, at a magnitude smaller than VNNI-targeted systems' eight-bit storage choice would suggest.