IFlux: Intent-Aware Storage Tiers & Software Scheduling for HPC Systems

Abstract

Modern High Performance Computing (HPC) workloads exchange datasets at high velocity to enable data-driven science. HPC systems employ hardware and software I/O accelerators to enable efficient data exchange. However, domain scientists and software engineers must manually request specific hardware accelerators and manually integrate software accelerators into their workloads, significantly hindering their adoption. Recently, scientists have proposed I/O intents, defined as “why”, “what”, and “how” users or applications perform certain I/O operations, that can be used to automatically configure storage systems for the user. We designed an intent-driven scheduler plugin called IFlux that uses the workload’s I/O intents to assist the HPC system scheduler to automatically and dynamically allocate hardware and software accelerators. In this work, IFlux advances the current state-of-the-art in three key aspects. First, IFlux automatically and efficiently maps different I/O intents to various hardware and software accelerators with a throughput of 3.5 M intents per second with a scheduler overhead of 0.02%. Second, IFlux provides a scheduler workflow that uses a scheduler’s job specification enhanced with intents to allocate the required storage accelerators, deploy necessary middleware software, and enable workloads to use the allocated hardware and deployed software transparently with a runtime overhead of 0.01%. Finally, IFlux speeds up popular benchmarks such as the IOR and DLIO Benchmarks, which represent six classes of workloads from simulation, data analytics, and artificial intelligence, by up to 147× for specific use cases. In conclusion, IFlux enhances existing HPC schedulers with an intent-driven approach to speed up large-scale HPC workloads such as the 1000 Genomes workflow and Megatron Deepspeed by up to 3.12× on an institutional cluster.