From Per-Job Data to an Aggregated Workload Insight: A Toolkit for Profiling HTCondor Workload
TL;DR
A Python toolkit that aggregates raw job data into four cluster-level views, designed to answer a specific question a researcher or facilitator would ask when triaging a workload, transforming thousands of individual job records into a concise, interpretable summary.
Abstract
HTCondor users for high-throughput computing often struggle to quickly understand how their computational workloads are performing. Current interfaces expose large volumes of raw job data, making it difficult to diagnose common issues such as jobs stuck on hold, poor resource utilization, or unexpected failures. These issues further snowball when dealing with large clusters. We present a Python toolkit, developed at the Center for High Throughput Computing (CHTC) at the University of Wisconsin–Madison, that bridges this gap. To be included as a part of the HTCondor suite, given a single cluster ID the toolkit aggregates raw job data into four cluster-level views: a status dashboard showing the distribution of job states, a runtime histogram revealing duration variance and flagging anomalously short runs, a hold classifier that groups held jobs by reason code with plain-language explanations, and a resource utilization report comparing requested versus actual CPU, memory, and disk usage. Each view is designed to answer a specific question a researcher or facilitator would ask when triaging a workload, transforming thousands of individual job records into a concise, interpretable summary.