Preprint
Sep 2026
HyDra: Demystifying and Taming Dynamic Context Parallelism at Production Scale
Long-context training runs on sequences whose lengths span orders of magnitude, and dynamic context parallelism (DCP) gives each sequence its own CP degree. Existing DCP systems either do not scale or perform poorly on mainstream models, leaving Megatron-Core (Mcore) DCP as the only option at production scale. Mcore DC...
Zi-Hao Fan, Yun-Zhuo Liu, Bo Jiang et al.
· 0 citations