As Mixture-of-Experts (MoE) models continue to scale, serving them increasingly relies on expert parallelism (EP) across a growing number of devices. Yet skewed expert workloads create imbalance across computation, communication, and memory, making load balancing a central optimization objective in distributed MoE serv...
Chang Chen, Andrew Yang, Tian-Cheng Chen et al.· 0 citations
This work systematize the RL-for-LLM paradigm and provides a compute-centric analysis of prominent post-training algorithmic frameworks: Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), as well as their variants, and develops a taxonomy of intra- and inter-model parallelism strategies for...
Maciej Besta, L. Schmidt, Lara Nonino et al.· 0 citations
Building on NCCL's device-side API, low-latency interfaces for constructing custom collective kernels are developed and used to implement new symmetric collectives in NCCL, demonstrating benefits for both AI inference and traditional HPC workloads.
Siyuan Shen, Anton Korzh, J. Bachan et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.