Dec 2026· IEEE Transactions on Parallel and Distributed Systems· Vol 37, pp. 2759-2775· 0 citations· 82 references
TL;DR
This work proposes CeeTrain, a geo-distributed model training system tailored for heterogeneous GPUs and networks in CEE environments, which adopts a communication-centric design philosophy to tackle challenges arising from slow and unstable inter-region networks.
Abstract
Most existing training systems focus on a single region. In contrast, we envision that cross-region training offers more flexible GPU resource allocation and yields significant potential. However, the hierarchical cluster topology and unstable networks in the cloud-edge-end (CEE) environment, a typical cross-region scenario, pose substantial challenges to building an efficient and autonomous model training system. We propose CeeTrain, a geo-distributed model training system tailored for heterogeneous GPUs and networks in CEE environments. CeeTrain adopts a communication-centric design philosophy to tackle challenges arising from slow and unstable inter-region networks. It begins with a heterogeneous device profiler that identifies and groups devices based on both network and compute characteristics. Leveraging device groups, CeeTrain implements compact, zero-bubble pipeline parallelism, automatically deriving optimal parallel strategies. To further adapt to runtime variability, CeeTrain integrates a dynamic environment adapter that reacts to network fluctuations. Extensive evaluations demonstrate that CeeTrain achieves 1.3-2.8× higher training throughput compared to widely used and SOTA training systems.
Training foundation models at the frontier typically requires large-scale dedicated and centralized clusters with fast interconnects - infrastructure that is prohibitively expensive and generally accessible only to well-resourced organizations. Decentralized training over pooled, geographically distributed GPUs offers...
B. Hoang, Tian-Zhang He, Mohammad Goudarzi· 0 citations
As the size of large language models (LLMs) continues to grow, training these models typically relies on data centers equipped with high-performance GPUs. However, in modern data centers, acquiring large-scale homogeneous GPU resources often results in long queueing delays, and training with on-demand instances incurs...
Chen-Hao Wang, Jun Wu, Ying-Hao Yu· IEEE Transactions on Network...· 0 citations
GeoMesh is presented, a synchronous geo-distributed training framework for heterogeneous GPUs that balances per-worker workloads by assigning each GPU a suitable batch size and number of inner steps, so faster GPUs do more useful work instead of waiting.
Chang-Yong Shin, Jaerim Park, Mi-Gyung Kang et al.· Conference on Empirical Meth...· 0 citations
The growing computational power of modern GPU data centers has let a few industrial giants train and deploy large language models (LLMs) on massive amounts of data. Yet the sheer scale of these models and their training data makes centralized development increasingly hard to sustain, and at the same time shuts out smal...
Zi-Yue Luo, Jiaxuan Cai, C. Le Denmat et al.· 0 citations
Chameleon is a synchronous and adaptive bidirectional pipeline framework that achieves high throughput in dynamic and heterogeneous edge systems, and incorporates a hardware-aware throughput model to capture pipeline stage performance on heterogeneous devices.
Tao Liu, Xiang-Peng Hou, Li-Ying Li et al.· International Conference on...· 0 citations
State-of-the-art deep learning models rely on large GPU clusters and various parallelism strategies, which in turn depend on collective communication (CC) operators to synchronize data. While vendor libraries (e.g., NCCL, RCCL) provide standard CC algorithms, they often suffer from bandwidth bottlenecks in modern, comp...
Chen-Yang Hei, Fu-Liang Li, Dong-Lin Zheng et al.· IEEE Transactions on Network...· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comOct 29, 2026
Host Sowbhagya Suresh shares some of the latest news and cutting-edge research from across Microsoft—from understanding the challenges facing coding agents to uncovering how people use AI at work and improving extended-range weather forecasting. The post Microsoft Research Focus – Edition #97 appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduOct 28, 2026
Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.
For decades, a fundamental question about the difficulty of computing in high-dimensional spaces has resisted proof. New research resolves this question, removing a major theoretical obstacle to understanding the efficiency of randomized algorithms used across science and computing. The Kannan–Lovász–Simonovits (KLS) conjecture was proposed in 1995. It asks whether, for a broad class of high-dimensional […] The post Solving a 31-year-old grand challenge in mathematics – the KLS conjecture appear…
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.