This work design agents to intelligently deliver the entire life-cycle of an HPC application experimental run in cloud -- creation and build of a container, deployment in Kubernetes, optimization, and orchestration of a scaling study.
Abstract
Large Language Models (LLMs) are serving as a catalyst of change for research practices, touching the daily lives of staff scientists, software engineers, and system administrators. The developments promise new degrees of autonomy, where categories of human work and decision making are replaced by autonomous, goal-oriented systems. This transition necessitates novel architectural paradigms and solid understanding of the strengths and limitations of LLMs. In this work, we design agents to intelligently deliver the entire life-cycle of an HPC application experimental run in cloud -- creation and build of a container, deployment in Kubernetes, optimization, and orchestration of a scaling study. We pursue this task for four well-known HPC applications to build multi-platform images and optimize across 21 instance types in Kubernetes. We demonstrate successful linear scaling with patterns approved by human experts, designs that improve work time to completion, and review suggested best practices for agentic design and collaboration.
The extent to which an agentic framework can optimize and run an HPC scaling study with a low latency network in Amazon Web Services, accurately transform HPC job specifications between workload managers, and design and run an entire biosciences workflow is assessed.
Vanessa V. Sochat, Daniel Milroy· arXiv.org· 0 citations
This paper presents a hierarchical, dynamic architecture and software to discover resources across diverse cloud, edge, and HPC systems and exemplifies the importance of careful coordination between agents, discovery tools, and infrastructure for agentic science.
The emergence of modern agents powered by large language models has created a demand for executing long-horizon, autonomous workflows in various domains that require significant computational resources. While High Performance Computing clusters provide the ideal infrastructure for these computation-intensive workloads, traditional HPC job schedulers such as Slurm are not designed for dynamic, agentic workflows characterized by unpredictable task durations, external API calls, and fault tolerance requirements of modern agents. This work presents RASER, a user-space framework that enables seamless execution of agentic workflows on production HPC clusters by extending Slurm's internal primitives. RASER introduces agentic job arrays with work stealing via shared filesystem queues, user-space checkpointing through application-level state serialization combined with Slurm requeue, and Apptainer container-based isolation without requiring any image modifications. Evaluations demonstrate that RASER reduces makespan by nearly 39% compared to static partitioning while achieving near-full CPU utilization. RASER provides resilience against preemption and failures while maintaining minimal checkpoint/restore overhead. It requires no kernel privileges or external database infrastructure, making it an accessible solution for deploying agentic workflows on existing HPC infrastructure.
EnsembleLauncher is introduced, a recursively hierarchical workflow orchestrator for exascale systems, featuring a fully decentralized control plane and a programmable scheduling policy interface that outperforms state-of-the-art tools by more than four times.
Harikrishna Tummalapalli, Christine M. Simpson, Riccardo Balin et al.· 0 citations
The paper addresses the transformation of enterprise application infrastructure out of on-premise legacy resource setting into service-based cloud environments properly configured to scale horizontally, and presents experimental evaluations of the response time, throughput, service resiliency, and infrastructure utilization in both traditional and cloud-native deployments.
Kanya Mohammed, Naree Thongchai· International Journal of Mod...· 0 citations
This work organizes agentic workflows in a taxonomy and presents its first architectural characterization with a production study at Microsoft Azure and a controlled study of open-source frameworks, showing that agentic execution is fragmented and heterogeneous.
Ji-Rong Yang, Peizhe Liu, Chaojie Zhang et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
