Modern High Performance Computing (HPC) workloads exchange datasets at high velocity to enable data-driven science. HPC systems employ hardware and software I/O accelerators to enable efficient data exchange. However, domain scientists and software engineers must manually request specific hardware accelerators and manually integrate software accelerators into their workloads, significantly hindering their adoption. Recently, scientists have proposed I/O intents, defined as “why”, “what”, and “how” users or applications perform certain I/O operations, that can be used to automatically configure storage systems for the user. We designed an intent-driven scheduler plugin called IFlux that uses the workload’s I/O intents to assist the HPC system scheduler to automatically and dynamically allocate hardware and software accelerators. In this work, IFlux advances the current state-of-the-art in three key aspects. First, IFlux automatically and efficiently maps different I/O intents to various hardware and software accelerators with a throughput of 3.5 M intents per second with a scheduler overhead of 0.02%. Second, IFlux provides a scheduler workflow that uses a scheduler’s job specification enhanced with intents to allocate the required storage accelerators, deploy necessary middleware software, and enable workloads to use the allocated hardware and deployed software transparently with a runtime overhead of 0.01%. Finally, IFlux speeds up popular benchmarks such as the IOR and DLIO Benchmarks, which represent six classes of workloads from simulation, data analytics, and artificial intelligence, by up to 147× for specific use cases. In conclusion, IFlux enhances existing HPC schedulers with an intent-driven approach to speed up large-scale HPC workloads such as the 1000 Genomes workflow and Megatron Deepspeed by up to 3.12× on an institutional cluster.
Hariharan Devarajan, Vanessa V. Sochat, Daniel Milroy et al.· IEEE Transactions on Paralle...· 0 citations
AI-driven scientific workloads increasingly depend on data-intensive input pipelines, where deep learning frameworks must ingest and transform large datasets from hierarchical HPC storage. Existing system-centric data services improve movement and locality between the parallel file system (PFS), node-local storage, and memory. However, they do not directly optimize how input pipeline operations execute across scopes, stage overlap, and resource-specific parallelism. As scale grows, this gap causes worker stalls, contention, and poor hardware utilization. We present WADO, a distributed write-once-read-many (WORM) object-store runtime for data-centric workloads that closes this gap through three coordinated mechanisms: scope-centric processing, explicit pipeline decomposition, and interference-aware explicit parallelism. WADO dynamically maps operations to execution scopes, overlaps stages such as I/O, communication, and transformations, and applies contention-aware concurrency control to match hardware behavior at runtime. Our evaluation shows three main findings: (1) scope-centric processing preserves throughput under scale, improving mixed-operation throughput by up to 1.65 × ; (2) explicit pipeline decomposition converts serialized wait into overlapped progress, delivering up to 2.16 × higher sustained bandwidth; and (3) interference-aware explicit parallelism improves effective bandwidth by up to 4.4 × by avoiding oversubscription collapse. On Unet3D model training, these mechanisms translate to end-to-end gains, improving data loading performance by 4.1 × compared to baseline PyTorch on Lustre, and 1.51 × compared to DYAD, enabled by deeper pipelining, adaptive parallelism, and near-data transformation offloading.
Karim Youssef, Hariharan Devarajan, Nikoli Dryden et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
