Justus Theile

Sub-Two-Bit Ternary Weight Storage with Table-Free Arithmetic Decoding for CPU Inference

Ternary (1.58-bit) language models are stored in practice at two bits per weight, 26% above the information content of a ternary symbol. The code that removes that redundancy is not new: five ternary values as the base-3 digits of one byte at 1.600 bits per weight is llama.cpp's TQ1_0 and Spectra 1.1's TQ1; the depth-o...