Aug 2026· 1 citation· ⚡ 1 influential· 31 references
Computer Science
TL;DR
This work evaluates whether Model FLOPs Utilization (MFU) can serve as a portable, software-defined predictor of GPU power for LLMs, finding that a linear MFU-based power model fits every tested GPU as long as the workload is compute-bound, as in production LLM training.
Abstract
High-fidelity performance simulators are essential for designing and configuring efficient AI systems, yet today's tools lack the ability to predict power consumption. Established GPU power models rely on hardware utilization counters, which do not exist until the workload has actually run. This work evaluates whether Model FLOPs Utilization (MFU)-an analytical, software-defined metric relating achieved throughput to peak hardware capability-can serve as a portable, software-defined predictor of GPU power for LLMs. We benchmark almost 3000 single-device training runs across six GPUs, covering different model families, numerical precisions, batch sizes, and context-window lengths. We find that a linear MFU-based power model fits every tested GPU as long as the workload is compute-bound, as in production LLM training. Fitting per-(GPU, dtype, batch size) instead of per-GPU drops the within-cell mean error from around 10% to around 1%, matching the cross-repeat measurement-noise floor.
Increasingly large deep neural networks (DNNs) pose a significant challenge regarding required compute capabilities and energy consumption, especially at the edge. This challenge generally necessitates dedicated inference hardware accelerators. Hardware/software co-design can improve overall system performance by jointly optimizing the hardware micro-architecture and DNN workload mapping. It is however not easy to quantify how well such an accelerator generalizes to workloads not considered during hardware/software co-design. In response to this, we introduce the Compatibility Ratio (CR) as a simple guideline for evaluating performance trade-offs between optimal hardware micro-architecture configurations across different workloads. CR allows us to quantify the performance trade-offs of deploying a workload on an accelerator optimized for a different workload. We demonstrate CR through two case studies on a systolic array-based accelerator. First, we apply CR to explore the design space of the accelerator across 13 DNN workloads. In this case study, CR analysis showed that the choice of representative workload during co-design can implicitly increase the normalized area-latency cost of unconsidered workloads by more than 30% in the evaluated design space. Furthermore, we use CR to analyze how well our systolic array-based accelerator template can generalize beyond a single DNN workload to cover a family of DNN workloads. Our findings show that, for the considered accelerator, a DNN model-family optimized configuration might occupy an effective middle ground between highly targeted single- and general-purpose configurations. Second, we use CR as a guideline for a practical memory-retargeting decision in a specialized variant of our accelerator template. In this case study, CR quantifies whether a memory-retargeted accelerator derivative is justified under the selected memory-area and latency objective. For this two-configuration memory-retargeting case, analytical CR differs from implementation-level CR by 0.01, corresponding to one percentage point on the normalized CR scale.
Lukas Groth, Andrija Nešković, Rainer Buchty et al.· ACM Transactions on Embedded...· 0 citations
Graphics Processing Units (GPUs) have been serving as critical computation resources for large-scale parallel computations. With increasing chip complexity, power efficiency has become an important design objective for modern GPUs. GPU power optimization relies on fast power evaluation, requiring architecture-level GPU power model. However, because of the time-consuming power label collection, only simple microbenchmarks are adopted for training. The limitation of microbenchmarks as training data incurs low accuracy for existing architecture-level GPU power models like AccelWattch. To address the limitation of microbenchmarks as training data, we propose G-Power, an architecture-level GPU power modeling framework that utilizes additional known GPU chips to provide additional knowledge. G-Power utilizes the aggregated knowledge foundation from additional known GPU chips and then performs fine-tuning on our target GPU. To provide foundations with additional known GPU chips and capture the similarity to utilize these foundations for fine-tuning, G-Power adopts a three-phase algorithm consisting of 1) pre-training with additional known chips, 2) attention-inspired aggregation, and 3) fine-tuning on our target GPU. We evaluate G-Power on four modern NVIDIA GPUs, demonstrating high accuracy. G-Power can achieve a low MAPE of 14% and a high correlation coefficient R of 0.88 on average, which are 22% lower MAPE and 0.36 higher R than AccelWattch.
Qijun Zhang, Yao Lu, Shang Liu et al.· 0 citations
The high-performance computing industry is moving beyond an era in which each generation of GPU provides uniform performance gains across all applications. The growing importance of AI is driving GPU architecture towards greater specialization, with more silicon devoted to Tensor Cores and reduced-precision arithmetic. As a result, the benefits of new chip generations are increasingly dependent on the characteristics and computational demands of the workload being run. This project analyzes how the architectural changes introduced across NVIDIA’s Volta, Ampere, and Hopper GPU generations translate into real-world performance and energy efficiency gains.
Experiments were conducted using one GPU per run to isolate device-level performance and avoid the effects of inter-GPU communication. The application suite included scientific computing, AI training, AI inference, and targeted microbenchmarks to stress different parts of the GPU such as matrix computation, memory bandwidth, and data movement. For each workload, we measured performance, execution time, energy consumption, and GPU hardware counters to compare application scaling with the architectural advancements introduced across each generation.
The results show that performance improvements from Volta to Hopper range from 2.95x to 8.32x, demonstrating that generational gains depend strongly on workload characteristics. Matrix-heavy applications, including many modern AI workloads, achieved the largest improvements in both performance and energy efficiency by effectively using increasingly capable Tensor Core hardware. In contrast, memory-bound and vector-based workloads experienced more modest gains because memory bandwidth and general-purpose compute resources have scaled more slowly.
Overall, the results show that specialization can deliver substantial performance and efficiency gains, but only when workloads can effectively use the hardware being added. Future work will extend the analysis to additional GPU generations and applications, while exploring frequency capping and more detailed die-area analysis.
Supervisor: Ian Karlin
Matthew Tindale, I. Karlin, Tobias Salamon et al.· Inquiry@Queen's Undergraduat...· 0 citations
SLIM (Saturation-Aware Lightweight Performance Model), a semi-analytical model that predicts LLM inference throughput and latency from analytical formulations of Transformer computation and memory traffic, is introduced, which outperforms representative performance-modeling baselines while successfully generalizing to previously unseen operating conditions.
Pol G. Recasens, F. Agulló, Yue Zhu et al.· arXiv.org· 0 citations
Three fundamental design principles are revealed that provide design-space guidance for architects designing the next generation of memory-accelerated LLM systems.
Corey Lammie, Hadjer Benmeziane, W. Simon et al.· 0 citations
A mathematical model is proposed to predict throughput and energy consumption for concurrently executing CV workloads on edge GPU accelerators and can be integrated into functional simulation frameworks for edge–cloud deployment studies.
Abhinaba Chakraborty, D. Colle, M. Pickavet et al.· Journal of Real-Time Image P...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
