Jul 2026
MatrixFSDP: communication-free matrix optimizers under ZeRO-3 parameter sharding
MatrixFSDP changes where ZeRO-3 shards live, not the optimizer being computed, to match full-matrix Muon while preserving ZeRO-3-scale memory, and reduces optimizer-step latency over stock FSDP2-Muon by 4.2x.
Ming Gao, Yanwu Xu, Hao Zhang
· arXiv.org · 0 citations