This work characterize the fine-tuning behavior of representative encoder-only SLMs of BERT variants, and autoregressive decoder-only SLMs of Pythia variants on GLUE benchmarks, and proposes a simple yet effective ML-based model selection that selects energy-optimal GPU DVFS settings on resource-constrained embedded platforms.
Abstract
Dynamic Voltage Frequency Scaling (DVFS) on resource-constrained embedded GPU platforms is essential for energy-efficient small language model (SLM) fine-tuning, as privacy- and personalization-driven adaptation increasingly requires local execution and involves repeated forward-backward optimization over many mini-batches, making it substantially more time- and energy-intensive than single-pass inference. To this end, 1) we first characterize the fine-tuning behavior of representative encoder-only SLMs of BERT variants, and autoregressive decoder-only SLMs of Pythia variants on GLUE benchmarks. In addition to the characterizations, 2) we propose a simple yet effective ML-based model selection that selects energy-optimal GPU DVFS settings on resource-constrained embedded platforms. Our results on NVIDIA Jetson AGX Orin demonstrate average 13.11% energy savings (up to 26.73%) over MAXN Mode 0, which has no explicit power cap.
Results show that compact SLMs paired with PEFT provide a practical, energy-aware path to personalized on-device deployment, with the optimal method set by the dominant constraint: LoRA+ for energy and QLoRA for memory.
This work evaluates whether Model FLOPs Utilization (MFU) can serve as a portable, software-defined predictor of GPU power for LLMs, finding that a linear MFU-based power model fits every tested GPU as long as the workload is compute-bound, as in production LLM training.
Existing mobile GPU dynamic voltage and frequency scaling (DVFS) policies rely on coarse-grained utilization metrics and treat the GPU as an isolated control domain, failing to reflect the layer-level computational and memory diversity of deep neural network (DNN) inference. This paper proposes PUMA, a performance monitoring unit (PMU)-guided multi-domain layer-aware DVFS framework. PUMA combines offline layer/layer-chain PMU characterization with online GPU PMU observations to identify execution characteristics associated with compute-bound, memory-bound, and bursty phases. In the offline stage, representative DNN workloads were profiled across 504 frequency combinations spanning the GPU, memory interface (MIF), and internal interconnect (INT) domains to derive PMU thresholds and domain-specific frequency-correction rules. At runtime, PUMA applies threshold-based bounded corrections to the GPU, MIF, and INT frequency decisions on top of the existing governors. PUMA was implemented at the kernel level on Google Pixel 9 and evaluated using six DNN workloads. Compared with the existing governor, PUMA reduced SoC power by 28.54% on average and by up to 33.41%, while reducing energy by 23.76% on average and by up to 27.97%, with an average inference latency increase of 7.28% and a maximum increase of 12.82%. Compared with GPU-only correction, full PUMA further reduces average power by 11.4% relative to GPU-only correction, and PUMA achieves a lower energy-delay product than the existing governor across all evaluated workloads.
W. Chang, Seung-Ryeol Ohk, Young-Jin Kim· IEEE Access· 0 citations
The development of long-context Large Language Models (LLMs) is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding. To overcome the inherent trade-offs between the memory overhead of metadata-based metrics and the computational inefficiency of adaptive selection strategies, we present Faster Flash Decoding (FFD), a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding. FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning via low-bit quantization. Furthermore, we introduce the top-delta strategy, which dynamically filters blocks to achieve distribution-adaptive sparsity without global synchronization. Offering a training-free and plug-and-play solution, FFD also enables the reuse of scanning results for computation, achieving up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement. Empirical validation on RULER and LongBench confirms that FFD maintains model accuracy while delivering high-ratio sparsity, with code available at https://github.com/qluoluo/faster-flash-decoding
Zhigeng Liu, Zhiyuan Ning, Ruixiao Li et al.· 0 citations
Design rules and a reproducible evaluation protocol are contributed that jointly report quality, memory, and end-to-end speed, and a foundation for automated pipeline search under realistic single-GPU constraints is provided.
Results show that energy-aware serving should jointly optimize both request energy and token energy, rather than only reducing per-token energy cost, and substantially narrows the dense-vs-MoE token-energy gap.
P. Vellaisamy, Vanessa Lam, Shawn Blanton et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
