Beyond Accuracy: Prefix-Invariant Realizations of Low-Precision Fast Matrix Multiplication
Fast matrix multiplication saves multiplications through exact cancellation, but rounding sums that mix token rows can leave contributions from later tokens in earlier language model outputs. This threatens prefix invariance, which multiple-choice likelihood scoring relies on: a scored likelihood must depend only on it...