Kaushik Kandadi

NCCL M2N: A Layout- and Topology-Aware Collective for Distributed Tensor Resharding

Distributed training and rollout generation often use different tensor layouts, requiring model weights to be resharded across distinct process groups. This M-to-N redistribution is not directly expressed by standard collectives. Flat direct sends duplicate traffic across destination replicas, while gather-then-broadca...