Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
BF16 is now standard in large-scale pretraining, including in fused attention kernels such as FlashAttention, and these kernels are widely trusted. When we used FlashAttention-3 to pretrain a 450M-parameter transformer on 50B tokens, however, we ran into a problem: training was healthy for 25B tokens, then the gradient...
Jun-Lin Chen, Daize Dong, Huan-Wei Di et al.
· 0 citations