Dynamic Compute and Network Orchestration for Disaggregated RL
TL;DR
This work builds Silverstone to orchestrate dynamically both compute and network in disaggregated RL, using a reconfigurable optical-electrical fabric called RFabric that achieves superior performance-cost efficiency at scale over static Fat-Tree networks.
Abstract
Disaggregating the generation and training stages in RL is widely adopted to scale LLM post-training. There are two critical challenges here. First, the generation stage often becomes a bottleneck due to dynamic workload shifts and severe execution imbalances. Second, the decoupled stages result in diverse and dynamic network traffic patterns that strain the conventional static fabric. We build Silverstone to orchestrate dynamically both compute and network in disaggregated RL. Silverstone employs an adaptive compute scheduler that adjusts parallelism configuration to match changing workload characteristics within and across generation steps. Silverstone adopts a reconfigurable optical-electrical fabric called RFabric: It leverages optical circuit switches to reconfigure the aggregation and core layers of the topology on demand, tailoring bandwidth resources to the unique communication patterns across various phases of training, generation, and weight synchronization. Evaluated on a 64-H800 GPU testbed, Silverstone demonstrates up to a 1.42× throughput improvement over static baselines. Using a high-fidelity simulator, we also show that RFabric achieves superior performance-cost efficiency at scale over static Fat-Tree networks.