HyScale: Scaling Long-Context Training for Learned Sparse Attention
HyScale, a long-context training system for learned sparse attention, introduces HyTopK and selection replay, and speeds up GLM-5.1 training on 1M-token sequences by 2.07$\times$ over Megatron-LM on 512 H20 GPUs.
Yang-Shen Deng, Xin-Jing Huang, Yulong Wang et al.
· 0 citations