Aug 2026· Journal of ICT Standardization· Vol 14, pp. 357-390· 0 citations
Computer Science
TL;DR
A standards-based, multi-tenant cloud inference framework that integrates OpenStack orchestration with Single Root I/O Virtualization (SR-IOV)-enabled graphics processing unit (GPU) partitioning to achieve predictable and isolated real-time inference execution.
Abstract
The deployment of real-time artificial intelligence inference services on shared cloud infrastructure poses significant challenges due to resource contention, latency variability, and tail-latency amplification. While cloud platforms offer scalability and flexibility, conventional accelerator sharing mechanisms often fail to provide the determinism required by latency-sensitive inference workloads. This paper presents a standards-based, multi-tenant cloud inference framework that integrates OpenStack orchestration with Single Root I/O Virtualization (SR-IOV)-enabled graphics processing unit (GPU) partitioning to achieve predictable and isolated real-time inference execution. In the proposed architecture, each tenant is assigned an exclusive GPU virtual function, enabling hardware-level isolation while remaining fully compatible with native OpenStack scheduling and resource management mechanisms. A comprehensive experimental evaluation is conducted on a private OpenStack cloud to assess inference latency distribution, tail behavior, scalability, robustness to background network and control-plane activity, and throughput-latency trade-offs. Experimental results show that median inference latency remains stable across single-tenant and multi-tenant configurations, while P95 and P99 tail latencies exhibit no measurable amplification under concurrent execution. The system scales linearly with the number of available GPU virtual functions, maintaining consistent latency behavior until hardware capacity is reached. Additional experiments demonstrate that background network traffic and control-plane operations introduce negligible impact on inference latency. Throughput analysis reveals a well-defined saturation knee, enabling clear identification of safe operating regions for real-time inference services. By leveraging mature ICT standards and open-source cloud infrastructure, this work provides a reusable reference architecture for deploying latency-sensitive inference services in private and hybrid clouds. The results highlight the effectiveness of hardware-assisted accelerator isolation in balancing performance determinism, scalability, and operational simplicity, and offer practical guidance for future system design and standardization efforts.
A structured, paired comparison of configuration complexity and performance across six platforms (Proxmox VM and LXC, native KVM, OpenStack VM and Zun, and Podman) on a single NVIDIA RTX 4500 Ada GPU, emphasizing the under-documented container paths
Priska Steininger, Manfred Pamsl, Helmut Lindner et al.· Computers· 0 citations
With the rapid growth of artificial intelligence (AI) and high-performance computing (HPC), GPUs and other accelerators have become a shared computing substrate for a wide range of workloads. However, many shared accelerator clusters still rely on coarse device-level allocation, which often leads to low effective utilization, resource fragmentation, and long queueing delays. Although pooling technologies offer a promising direction, existing approaches remain limited in supporting fine-grained, low overhead sharing across heterogeneous accelerators and diverse co-located workloads. This paper presents gPooling, a hardware-agnostic accelerator pooling framework based on driver-level interception. gPooling creates elastic virtual devices on demand and extends fine-grained sharing across heterogeneous accelerators through a unified control path. We evaluate gPooling using benchmarks derived from real cluster traces and through deployment in a production GPU cluster. Results show that gPooling improves accelerator utilization, reduces user waiting time, and increases the overall efficiency of shared accelerator environments.
Kaicheng Guo, Jingyi Chen, Chen Chen et al.· IEEE Transactions on Paralle...· 1 citation
ElastiCo is presented, an elastic co-location framework that enables training and inference workloads to safely share GPUs through three integrated mechanisms that decomposes the resulting multi-resource allocation problem into per-job configuration selection subproblems via dynamic per-resource shadow prices.
Jing-Hao Wang, Yi-Hang Zhou, Xiaoyang Sun et al.· 0 citations
Moving from quantum research and development to production-grade, fault-tolerant quantum workload execution remains one of the most significant challenges facing quantum platform builders. While Python frameworks have enabled an easy entry point for quantum algorithm design, the low-latency requirements for real-time quantum error correction (QEC) demand performance that traditional interpreted environments cannot provide. FPGAs and ASICs play a central role at these layers, but their specialized programming models make development rigid and time-consuming. CPUs, GPUs, and other accelerators introduce a different challenge: as infrastructure becomes increasingly heterogeneous, programming across different devices and their associated abstractions becomes more complex. Allowing researchers to write workloads in high-level languages that map to low-latency execution across diverse distributed target platforms will enable the development of key infrastructure for utility-scale quantum systems. For this, we introduce $\textit{Backline}$, a heterogeneous compilation and runtime framework built within PennyLane and Catalyst. Backline allows us to design and build quantum-classical workloads for high-performance and low-latency devices, with compilation directly from a Python interface through MLIR. We demonstrate the compilation and execution of several quantum workloads with low-latency data movement across a mix of CPUs, GPUs, and FPGAs, for both local and distributed remote hardware targets, all from a vendor-agnostic Python frontend. With an AMD VPK120 FPGA board as the controller, issuing each round from its hardware-handshake engine, we measured median steady-state round-trip latencies over RoCE v2 of $2.305~\mu$s to an AMD Ryzen Threadripper PRO CPU and $4.5~\mu$s to an AMD Instinct MI210 GPU across $10^6-1$ rounds per path, demonstrating microsecond-scale synchronous co-processing.
Joseph K. L. Lee, M. Malekmohammadi, Hong-Sheng Zheng et al.· 0 citations
Industrial edge platforms increasingly consolidate real-time control and general-purpose workloads on a single system-on-chip (SoC) to reduce costs, power, and complexity. Conventional real-time virtual machine (RTVM) setups, however, typically rely on PREEMPT_RT Linux and commonly reserve at least two CPU cores to isolate real-time tasks from housekeeping, which hinders scalability on resource-constrained edge platforms. This paper investigates whether low-latency Ethernet networking with tight observed tail latency can be sustained in a single-core RTVM configuration. We implement a single-core Zephyr-based RTVM on ACRN and compare it with a single-core PREEMPT_RT Linux RTVM under the same VM topology, using the same directly assigned Intel i226-LM PCIe Ethernet NIC via passthrough and an identical UDP echo workload. Latency measurements across 30 million packets at a traffic rate of 8 thousand packets per second (8 kpps) characterize both averagecase and extreme-tail behavior. PREEMPT_RT Linux shows severe tail amplification even without interference (99.999th percentile: $\mathbf{2 1 6 3} \boldsymbol{\mu} \mathbf{s}$; max: $\mathbf{6 8 7 2} \boldsymbol{\mu} \mathbf{s})$, while Zephyr maintains a tightly concentrated latency distribution (99.999th percentile: $79 \mu \mathrm{s}$; max: $82 \mu \mathrm{s})$. Under full-system noisy-neighbor load, Zephyr preserves sub- $\mathbf{1 0 0}-\boldsymbol{\mu} \mathbf{s}$ observed tail latency, whereas Linux degrades further. These empirical findings provide evidence that a specialized RTOS-based RTVM can sustain tight tail-latency performance and low delay variation on a single core, challenging the conventional two-core provisioning strategy for real-time edge systems.
Chwee-Lin Choong, Yoong Siang Song, Xin Yuan et al.· IEEE International Conferenc...· 0 citations
Ghost is an OS-level GPU virtualization layer integrated directly into the open-source GPU driver, using a GPU container abstraction with cgroup -like APIs for compute and memory control and privileged hardware-level scheduling and preemption for dynamic compute resource management.
Unknown authors· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
