The efficient and scalable integration of quantum resources into high-performance computing (HPC) environments requires standardized mechanisms for resource management, scheduling, and workflow orchestration across diverse and heterogeneous infrastructures. The Quantum Resource Management Interface (QRMI) addresses this challenge through a thin, vendor-agnostic middleware layer that provides standardized APIs for scheduling, executing, and monitoring quantum workloads while exposing quantum resources as first-class schedulable resources alongside CPUs and GPUs. Although previous work demonstrated QRMI integration with the Slurm workload manager, its applicability across other workload managers remained unexamined. This paper extends the validation of QRMI to a broad range of workload managers, including PBS, LSF, Grid Engine, Kubernetes, and the Flux Framework, encompassing traditional batch schedulers, a cloud-native orchestration platform, and a graph-based scheduler. We examine the integration patterns, implementation requirements, and scheduler-specific considerations associated with each environment and compare QRMI with alternative approaches to quantum resource integration. We demonstrate that QRMI provides a portable and flexible abstraction layer that minimizes scheduler-specific modifications while enabling consistent access to heterogeneous quantum resources across both on-premises and cloud environments.
Thomas Badts, T. Boyle, Claudio Carvalho et al.· arXiv.org· 3 citations
Modern High Performance Computing (HPC) workloads exchange datasets at high velocity to enable data-driven science. HPC systems employ hardware and software I/O accelerators to enable efficient data exchange. However, domain scientists and software engineers must manually request specific hardware accelerators and manually integrate software accelerators into their workloads, significantly hindering their adoption. Recently, scientists have proposed I/O intents, defined as “why”, “what”, and “how” users or applications perform certain I/O operations, that can be used to automatically configure storage systems for the user. We designed an intent-driven scheduler plugin called IFlux that uses the workload’s I/O intents to assist the HPC system scheduler to automatically and dynamically allocate hardware and software accelerators. In this work, IFlux advances the current state-of-the-art in three key aspects. First, IFlux automatically and efficiently maps different I/O intents to various hardware and software accelerators with a throughput of 3.5 M intents per second with a scheduler overhead of 0.02%. Second, IFlux provides a scheduler workflow that uses a scheduler’s job specification enhanced with intents to allocate the required storage accelerators, deploy necessary middleware software, and enable workloads to use the allocated hardware and deployed software transparently with a runtime overhead of 0.01%. Finally, IFlux speeds up popular benchmarks such as the IOR and DLIO Benchmarks, which represent six classes of workloads from simulation, data analytics, and artificial intelligence, by up to 147× for specific use cases. In conclusion, IFlux enhances existing HPC schedulers with an intent-driven approach to speed up large-scale HPC workloads such as the 1000 Genomes workflow and Megatron Deepspeed by up to 3.12× on an institutional cluster.
Hariharan Devarajan, Vanessa V. Sochat, Daniel Milroy et al.· IEEE Transactions on Paralle...· 0 citations
This work assesses the reliability of a dispatch agent across 432 runs, and finds that descriptive metadata increases successful execution from 48% to 87% of 220 submitted jobs, eliminating architecture mismatch, and improving performance for five of ten measurable applications by up to 3.3x.
Fluence, a Kubernetes scheduler plugin backed by the Fluxion graph-based scheduler, enabling gang-scheduled placement for quantum-classical workloads and custom resources and shows that quantum-awareness can be added to a cloud-native scheduler without modifying user containers.
Vanessa V. Sochat, Daniel Milroy· arXiv.org· 2 citations
The extent to which an agentic framework can optimize and run an HPC scaling study with a low latency network in Amazon Web Services, accurately transform HPC job specifications between workload managers, and design and run an entire biosciences workflow is assessed.
Vanessa V. Sochat, Daniel Milroy· arXiv.org· 0 citations
This work design agents to intelligently deliver the entire life-cycle of an HPC application experimental run in cloud -- creation and build of a container, deployment in Kubernetes, optimization, and orchestration of a scaling study.
This paper presents a hierarchical, dynamic architecture and software to discover resources across diverse cloud, edge, and HPC systems and exemplifies the importance of careful coordination between agents, discovery tools, and infrastructure for agentic science.
Vanessa V. Sochat, Daniel Milroy· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
