AdapBench is the first approach which dynamically makes at-runtime decisions for prioritizing the most informative benchmarks and early-stopping each one once enough evidence has accumulated to accept or reject a performance change hypothesis.
Abstract
Cloud-based software systems frequently release new versions, a fact further accelerated through agentic coding. Each new release, however, brings the risk of performance regressions which can irritate end users, increase infrastructure costs, and increase carbon footprints. Therefore, developers strive to catch such regressions in testing stages, usually through so-called continuous benchmarking. The standard way of doing this is using microbenchmark suites which, however, take notoriously long to execute. Optimizing the execution of microbenchmark suites is, hence, an active field of research. In this paper, we propose AdapBench, an online scheduling system for reducing the execution duration of microbenchmark suites. AdapBench is the first approach which dynamically makes at-runtime decisions for prioritizing the most informative benchmarks and early-stopping each one once enough evidence has accumulated to accept or reject a performance change hypothesis. We evaluate our prototype on eight popular Java projects, commonly used by microbenchmark research, and four additional Go projects. At matched 90% change detections and false-positive rate compared to an optimistic uniform run, AdapBench saves 42% of the aggregate execution duration on the Java projects, with negligible scheduling overhead (below 0.001% of the total runtime). On the Go projects, AdapBench finds every performance change with zero false positives while saving 85% of the full run.
AddressSanitizer (ASan) is widely deployed in industrial C/C++ development, but its runtime and memory costs can limit routine use in continuous integration and production-like testing. Although many optimized ASan-like sanitizers have been proposed, heterogeneous workloads, configurations, and baselines make their pra...
Jia-Xin Xi, Cheng Wen, Zhi-Wei Lin et al.· Proceedings of the 41st IEEE...· 0 citations
Large organizations carry a long tail of infrastructure migrations that are worth doing but rarely urgent enough to compete with feature work, because each one costs significant engineering time per repository. This paper reports on our industrial experience from Automated Uplift, a multi-month initiative applying LLM-...
Mario Corchero, David Williams, Laszlo Kiss Kollar et al.· 0 citations
LMTracer is presented, a fine-grained and real-time performance profiling framework for production LLM services that embed profiling logic into the execution through graph-embedded probing and streaming buffered profiling data to CPUs on demand to keep the execution of user kernels uninterrupted.
Wei Liu, Yong-Chao He, Bo-Han Zhao et al.· Proceedings of the ACM SIGOP...· 0 citations
As large language models (LLMs) keep growing in size and complexity, their training frameworks evolve at a rapid pace as well. Therefore, continuous integration (CI) is critical for maintaining the quality and stability of these frameworks. However, unlike traditional software, CI for LLM training frameworks relies on...
Tian-Shuo Qiao, Nai-Qian Zheng, Xiao-Peng Liu et al.· 0 citations
Full execution of Static Application Security Testing (SAST) on every commit in a modern CI/CD pipeline creates significant bottlenecks, especially in Jamstack architecture characterized by high commit frequency and fragmented code changes. This study evaluates the effectiveness of the Selective-SAST approach based on...
Fadilah Tun Hazimah, Muhammmad Faisal, Emil Agus Salim Habi Talib et al.· JOUTICA· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.