Attention Quantization for Tabular Foundation Models
This work develops a quantization strategy for queries, keys, and values to FP8 and uses explicit FP8 matrix multiplication instructions to speed up the attention calculation and finds that it is crucial to align the quantization error in the test rows with the quantization error in the training rows, as otherwise the...