Aug 2026· Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication· pp. 46-59· 0 citations· 56 references
Computer Science
TL;DR
Evaluation on a 5-node GPU cluster under synthetic and production-derived churn shows that Connex reduces P99 tail spikes by up to 85% compared to NCCL-based baselines, achieves sub-second cutover, and maintains 100% goodput at moderate loads where baselines collapse to 0–28%, while incurring less than 5% steady-state overhead.
Abstract
Modern LLM serving systems increasingly adopt elastic inference pipelines where stages frequently join, leave, and migrate across nodes. However, existing GPU communication frameworks like NCCL assume static topologies, causing routing failures and P99 latency spikes during worker transitions that violate sub-millisecond tail latency requirements. We present Connex, a communication system that elevates endpoint mobility from exceptional failure to first-class primitive. Rather than optimizing individual mechanisms in isolation, Connex defines a mobility contract that the communication layer enforces whenever workers join, leave, or migrate while token streams, activations, or KV transfers are in flight. The contract is realized through three cooperating mechanisms: (1) epoch-based routing that bounds staleness without global coordination, (2) explicit handover protocols that preserve stream ordering and provide exactly-once delivery across migrations, and (3) credit-based backpressure with traffic-class isolation that prevents churn-induced interference with latency-critical paths. Evaluation on a 5-node GPU cluster under synthetic and production-derived churn shows that Connex reduces P99 tail spikes by up to 85% compared to NCCL-based baselines, achieves sub-second cutover, and maintains 100% goodput at moderate loads where baselines collapse to 0–28%, while incurring less than 5% steady-state overhead.
This work presents ARK, a distributed elephant-flow path reservation mechanism that coordinates senders to choose source ports whose hashes map concurrent flows onto distinct spines, without requiring switch changes or receiver-side packet reordering.
Hung-Chun Lin, Ting-Wei Hsu, Chung-En Ho et al.· Conference on Applications,...· 0 citations
Modern collective communication libraries (CCLs) execute a collective communication task (CCT) by decomposing it into multiple sub-tasks, each mapped to a specific Virtual Topology (VT), which is an ordered graph of GPUs (e.g., a ring or a tree), to maximize parallelism and link utilization. As AI training scales to larger clusters, network anomalies (congestion and failures) are unavoidable, and a single straggling VT can delay the entire CCT. Existing solutions either rely on low-level transport-layer solutions which lacks a cross-sub-task perspective, or static CCL scheduling, failing to adapt to the dynamic and heterogeneous networks. We present PReCCL, a drop-in NCCL replacement that combines software inband telemetry with cross-VT workload reallocation. Unlike hardware-dependent telemetry (e.g., P4-INT), PReCCL implements in-band monitoring within the CCL. It precisely measures the stall counts of each VT, and piggybacks the telemetry meta-data on existing collective traffic. With this fine-grained visibility, PReCCL employs a dynamic cross-VT reallocation mechanism that shifts workloads from straggler VTs (suffering from congestion or faults) to healthy ones. Reallocation occurs only at CCT boundaries using an epoch-based deterministic protocol, avoiding cross-rank consistency races. We implement PReCCL in NCCL and evaluate it on a 32-GPU testbed, large-scale simulation, and a 1024-GPU production cluster. On the testbed, PReCCL improves CCT completion by up to 2.1X and end-to-end training by 1.21X, with low overhead above tuned message-size thresholds.
Zhiyong Chen, Kaihui Gao, Li Chen et al.· Conference on Applications,...· 0 citations
DeltaServe is presented, a host-agnostic co-serving design that converts this idle inference capacity into LoRA fine-tuning throughput while preserving inference service-level objectives (SLOs).
Jiaxuan Chen, Jianshu She, Ye Yuan et al.· arXiv.org· 1 citation
This work presents Pallas, a \textit{proactive} KV-cache migration framework that prepares the inference state at the predicted target before handover, in parallel with ongoing source-side inference and token delivery.
TETHER combines persistent virtual IPs, anticipatory state migration, and transport continuity mechanisms to transparently relocate application state between satellites before performance degradation occurs, showing that anticipatory migration preserves long-lived TCP sessions across satellite transitions and enables applications to trade latency tolerance against migration overhead.
Emi Digby, Nishanth R. Sastry· Conference on Applications,...· 0 citations
This work introduces CSIG, a protocol that delivers precise, multi-bit bottleneck congestion signals via a fixed-length Ethernet header, and proposes Fast Ramp-Up, a congestion control primitive that leverages these bottleneck signals to reduce median RPC latency by 20% and unclaimed bandwidth by 60% in production.
Abhiram Ravi, Nandita Dukkipati, Weiwu Pang et al.· Conference on Applications,...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
