ElastiCo: Elastic Configuration and Interference-Aware Orchestration for GPU Clusters

Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs

Festina is presented, a profiling-guided, power-aware control plane to minimize cluster-wide energy for serverless LLM serving and performs energy-aware workload consolidation to reduce GPUs'static power consumption via SLO-aware migration.

Breaking the Tradeoff: Elastic and Isolated GPU Sharing with Ghost

Parallax: Performance Prediction for Training–Inference Co-Execution

GPU clusters for deep learning (DL) workloads, especially inference, generally suffer from low utilization due to resource over-provisioning for satisfying their strict latency requirements. Co-locating throughput-oriented training jobs with latency-sensitive inference services is a promising approach to reclaim idle resources. However, accurately predicting the performance interference of such heterogeneous workloads under two mainstream GPU sharing mechanisms—Time-Slicing and Multi-Process Service (MPS)—remains a critical challenge. Existing predictors either focus on single-tenant scenarios or lack the fidelity to capture the complex contention patterns between training and inference. In this paper, we present Parallax, a fine-grained and accurate performance prediction framework tailored for training–inference co-execution. Parallax introduces interpretable modeling strategies for the two primary GPU sharing paradigms. For Time-Slicing, we propose a simulation-based model leveraging the virtual time slice and switching overhead abstractions to reconstruct operator-level interleaving and context-switching costs. For MPS, we develop a two-stage framework that first predicts resource utilization under concurrency and then quantifies performance degradation caused by microarchitectural contention, such as memory bandwidth and cache. This resource-centric approach ensures robust generalization to unseen workload combinations. Extensive evaluations on modern GPUs demonstrate the high accuracy of Parallax, predicting execution latency with a MAPE of 4.33% for Time-Slicing and 6.12% for MPS across diverse DL models (averaged over training and inference). Parallax is available at https://github.com/HIT-CeeCG/Parallax.

Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs

This paper aims to enable computation- and communication-efficient GPU sharing across devices within local area networks (LANs), facilitating ubiquitous AI inference on heterogeneous personal devices. We achieve distributed task offloading via CUDA API remoting. However, beyond raw computation, network constraints emerge as the primary bottleneck: limited bandwidth, high-frequency API invocations, and cross-task contention significantly hinder performance. To address these challenges, we propose Gleam, a novel and network-efficient framework for task-generic GPU sharing across local-area CUDA devices, with three key contributions. First, we reduce bandwidth overhead in CUDA API remoting through automatic model weight caching, and mitigate accumulated latency from frequent API calls by asynchronous execution. Second, we design a runtime task scheduler that dynamically determines API remoting pairs between LAN clients and servers, explicitly accounting for both network conditions and GPU resource contention under parallel workloads. Finally, we introduce dedicated mechanisms to ensure CUDA context consistency across distributed executions. Extensive experiments on heterogeneous NVIDIA GPUs and diverse AI workloads show Gleam consistently outperforms state-of-the-art baselines, achieving 1.4-24.2 times improvements in API remoting efficiency and up to 1.79 times higher system throughput.

gPooling: An Elastic GPU Resource Management Framework for On-Demand Virtualization in Shared Accelerator Clusters

With the rapid growth of artificial intelligence (AI) and high-performance computing (HPC), GPUs and other accelerators have become a shared computing substrate for a wide range of workloads. However, many shared accelerator clusters still rely on coarse device-level allocation, which often leads to low effective utilization, resource fragmentation, and long queueing delays. Although pooling technologies offer a promising direction, existing approaches remain limited in supporting fine-grained, low overhead sharing across heterogeneous accelerators and diverse co-located workloads. This paper presents gPooling, a hardware-agnostic accelerator pooling framework based on driver-level interception. gPooling creates elastic virtual devices on demand and extends fine-grained sharing across heterogeneous accelerators through a unified control path. We evaluate gPooling using benchmarks derived from real cluster traces and through deployment in a production GPU cluster. Results show that gPooling improves accelerator utilization, reduces user waiting time, and increases the overall efficiency of shared accelerator environments.

Theseus: Runtime-Adaptive GPU Collective Communication with Hot-Swappable Schedules

Current GPU Collective Communication Libraries (CCLs) employ predefined schedules optimized for stable environments. Their supported schedules and selection logic are fixed at communicator initialization, which fails to account for evolving runtime conditions, such as workload characteristics and hardware health status. Consequently, long-running GPU jobs experience suboptimal performance after hours or days of execution, which translates into longer job completion times and wasted GPU cluster resources. To address this problem, we present Theseus, a novel CCL backend that provides schedule-level runtime adaptivity. It admits user-defined schedules and selection policies. As runtime conditions change, Theseus selects suitable schedules using cluster-wide runtime attributes beyond CCL-internal metrics. Moreover, it hot-swaps from the previous schedule consistently across GPUs with low overhead. Theseus acts as a drop-in replacement to facilitate integration. We evaluate Theseus extensively on various GPU workloads with intuitive policies. Compared with NCCL, Theseus achieves up to 1.61X speedup of communication time in stable environments and 2.46X in dynamic environments. It improves end-to-end job completion time by up to 1.84X while incurring comparable or lower overhead.