A.X K2 Technical Report
To support long contexts efficiently, Sparse Gated Attention (SGA), which combines sparse attention with gated attention, and adopt Gated Norm (GN) to stabilize large-scale training is introduced, which keeps 4-bit NVFP4 serving within one point of FP8 accuracy.
Cheolseung Baek, Dhammiko Arya, Eunki Kim et al.
· 0 citations