DVFS-Aware Energy-Efficient Implementation of DNNs on Heterogeneous Hybrid Parallel Computers

Abstract

High-performance heterogeneous computing has emerged as a key resource for training increasingly Deep Neural Networks (DNNs). Achieving optimal performance and energy efficiency on such platforms, however, requires careful resource optimisation at both application and platform levels. While state-of-the-art DNN optimisation approaches typically focus on only one of these dimensions or are designed for homogeneous platforms, this paper considers both application- and platform-level configurations for the bi-objective optimisation of DNN training, to minimise execution time and energy consumption on single-node heterogeneous computing platforms. After formulating the bi-objective optimisation problem, we propose a methodology to model the performance and dynamic energy profiles of DNN applications, expressed as functions of the number of batches and Dynamic Voltage and Frequency Scaling (DVFS) states. These models are then exploited by a model-based algorithm that employs batch distribution (application-level) and DVFS states (platform-level) as decision variables to solve the bi-objective optimisation problem. The algorithm produces execution time-energy consumption trade-off solutions in the form of Pareto fronts. The proposed approach is evaluated on two heterogeneous hybrid servers comprising multicore CPUs and many-core GPUs. Experimental results demonstrate that solutions incorporating integrated application- and platform-level configurations outperform those that consider only a single dimension, in terms of both performance and energy efficiency. Furthermore, comparing our approach with TensorFlow data parallelism and homogeneous distribution strategies demonstrates its superiority. A comprehensive analysis of the results reveals that DNNs achieve greater improvements in energy efficiency than in performance.