Aug 2026· Asia-Pacific Workshop on Networking· 1 citation· 33 references
Computer Science
TL;DR
OptiFlow is presented, among the first LLM-driven frameworks for automated design of high-performance collective communication algorithms, with key insight is a two-layer decomposition: the LLM generates compact data-movement intent expressed in a domain-specific language, while deterministic scheduling algorithms compile these programs into executable schedules.
Abstract
Training trillion-parameter models has made collective communication a dominant bottleneck in GPU clusters. Existing approaches face a fundamental tension: optimization-based synthesis techniques suffer from combinatorial complexity that can require minutes to hours, while analytic heuristics rely on rigid assumptions that often fail under production variability. To the best of our knowledge, we present OptiFlow, among the first LLM-driven frameworks for automated design of high-performance collective communication algorithms. Our key insight is a two-layer decomposition: the LLM generates compact data-movement intent expressed in a domain-specific language, while deterministic scheduling algorithms compile these programs into executable schedules. We further employ an iterative pipeline that uses real-hardware feedback to refine LLM-generated proposals, without updating model parameters or relying on manually designed search heuristics. Evaluated on a 32-GPU NVIDIA A100 cluster, OptiFlow discovers All-Gather schedules that outperform NCCL by up to 3.75 × , while it also surpasses TACCL and TE-CCL with gains up to 5.12 × and 3.28 × , and shows better optimization-time scalability.
Evaluating leading frontier and open-source code generation models on both intra-node NVLink and inter-node RDMA platforms reveals that even the strongest model, GPT-5.5, correctly implements and achieves competitive performance on only 30.7\% of the benchmark tasks.
Shuang Ma, Yu-Yi Li, Yihan Zhang et al.· 0 citations
LazyTrain is proposed, an optimization layer over a layer-streaming executor that formulates checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, then executes the solved policy during training.
Xiao-Jun Wu, Cehao Yang, Honghao Liu et al.· 0 citations
UniEP fuses the MoE communication and computation into MegaKernels, effectively transforming complex architectural tuning into a unified parameter search space for automated adaptability and incorporates a deterministic token ordering mechanism that guarantees numerical consistency with sequential execution, even under aggressive overlap schedules.
Size Zheng, Xuegui Zheng, Li-Wen Chang et al.· IEEE International Symposium...· 1 citation
EPIC mitigates imbalance via performance-aware expert migration and runtime expert activation, and then improves communication with topology-adaptive transport kernels and fine-grained computation-communication overlap.
Jiamin Cao, Qingxu Li, Yaozhong Liu et al.· Conference on Applications,...· 0 citations
Modern collective communication libraries (CCLs) execute a collective communication task (CCT) by decomposing it into multiple sub-tasks, each mapped to a specific Virtual Topology (VT), which is an ordered graph of GPUs (e.g., a ring or a tree), to maximize parallelism and link utilization. As AI training scales to larger clusters, network anomalies (congestion and failures) are unavoidable, and a single straggling VT can delay the entire CCT. Existing solutions either rely on low-level transport-layer solutions which lacks a cross-sub-task perspective, or static CCL scheduling, failing to adapt to the dynamic and heterogeneous networks. We present PReCCL, a drop-in NCCL replacement that combines software inband telemetry with cross-VT workload reallocation. Unlike hardware-dependent telemetry (e.g., P4-INT), PReCCL implements in-band monitoring within the CCL. It precisely measures the stall counts of each VT, and piggybacks the telemetry meta-data on existing collective traffic. With this fine-grained visibility, PReCCL employs a dynamic cross-VT reallocation mechanism that shifts workloads from straggler VTs (suffering from congestion or faults) to healthy ones. Reallocation occurs only at CCT boundaries using an epoch-based deterministic protocol, avoiding cross-rank consistency races. We implement PReCCL in NCCL and evaluate it on a 32-GPU testbed, large-scale simulation, and a 1024-GPU production cluster. On the testbed, PReCCL improves CCT completion by up to 2.1X and end-to-end training by 1.21X, with low overhead above tuned message-size thresholds.
Zhiyong Chen, Kaihui Gao, Li Chen et al.· Conference on Applications,...· 0 citations
LLMVisor is presented, a roofline-guided latency attribution model that captures the memory-bound and compute-bound phases via a concise piecewise-linear form over features proportional to FLOPs and memory I/O traffic and runs efficiently at microsecond scale.
Shuowei Jin, Xueshen Liu, Jiaxin Shan et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
