Jul 2026· Practice and Experience in Advanced Research Computing· 0 citations· 15 references
Computer Science
TL;DR
This study presents a cross-platform, multi-model empirical study, where several important observations are brought, including the contrastive effect of quantization under different hardware bottlenecks, along with a quantification of runtime delays caused by the lack of parallelism in the ARM architecture.
Abstract
Prefill or prompt processing and Decode or token generation are two distinct subphases of LLM inference that are greatly influenced by LLM accelerators such as GPT-Generated Unified Format (GGUF Q4_K_M), NormalFloat 4-bit (NF4) Quantization, FlashAttention-2 and others. Although these accelerators clearly improve end-to-end LLM inference performance, their effectiveness over these subphases remains largely understudied. To address this gap, we present a cross-platform, multi-model empirical study, where we deploy multiple ∼ 1B-parameter LLMs on GPU, CPU, and Raspberry Pi 4B edge hardware platforms in the presence and absence of these accelerators. Each test case evaluates 10,000+ inference runs with separate phase-wise and end-to-end performance indicators. Our study brings several important observations, including the contrastive effect of quantization under different hardware bottlenecks, along with a quantification of runtime delays (up to +139%) caused by the lack of parallelism in the ARM architecture. Based on these benchmarking results and observations, we identify several open research challenges in the concluding section. Our work is fully reproducible and open-sourced on GitHub1.
EdgeXpert is proposed, a software-hardware co-designed LLM accelerator that resolves this incompatibility and achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, while maintaining near-baseline accuracy.
Sangwoo Ha, Hyunwoo Seo, Yurim Jo et al.· 0 citations
Design rules and a reproducible evaluation protocol are contributed that jointly report quality, memory, and end-to-end speed, and a foundation for automated pipeline search under realistic single-GPU constraints is provided.
Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires balancing quality, latency, model footprint, and energy. This paper presents a controlled measurement study of self-hosted LLM inference across edge and near-edge deployment nodes: an NVIDIA Jetson AGX Orin and a near-edge server with CPU-only and GPU-enabled inference modes. We evaluate multiple open-weight LLMs and quantization variants using a fixed question-answering workload, and compare them against GPT-4o as a cloud-hosted accuracy and latency reference. Our benchmarking pipeline reports accuracy, model footprint, per-token decoding latency, prefill latency, and overall execution energy. The results show that GPU-enabled server execution provides the lowest compute-side latency, while Jetson Orin shows lower measured energy, consistent with its lower platform power under our setup. CPU-only execution is consistently dominated in latency for our workload and shows higher measured energy. We also show that parameter count and downloaded weight-file size alone do not reliably predict observed accuracy or latency. Finally, using Pareto-frontier analysis, we study how deployment decisions may change under possible streamed-token delivery overheads, highlighting that compute-side inference metrics alone can lead to suboptimal placement for latency-sensitive interactive web services.
Maysam Khatib, Moysis Symeonides, Demetris Trihinas et al.· 0 citations
The operational energy consumption of large language model (LLM) inference is becoming an increasingly important component of the environmental footprint of deployed AI systems. However, direct measurement of inference energy often requires hardware telemetry, power instrumentation, or infrastructure-specific monitoring, limiting its applicability in comparative studies, early-stage system design, and sustainability reporting. This report presents an analytically structured, empirically calibrated, GPU-level methodology for estimating LLM inference energy on NVIDIA H100-class accelerators without direct runtime measurement. The proposed estimator combines parameter-scaled transformer FLOP accounting, calibrated memory-traffic factors, and hardware-specific energy coefficients for FP16/BF16 tensor-core computation and high-bandwidth-memory movement. It explicitly separates prompt prefill from autoregressive decoding, enabling energy estimates for input tokens, output tokens, and complete inference requests. The methodology further decomposes total energy into compute, parameter-access, key-value-cache write, and attention-read components, allowing the scaling behavior with model size, context length, and generated-token count to be analyzed. The resulting estimates are not intended to replace physical power measurements; rather, they provide transparent, reproducible, and assumption-explicit approximations suitable for model comparison, green-coding analysis, and design-time evaluation of LLM inference workloads.
Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis et al.· arXiv.org· 1 citation
Hydra is presented, a common-schema, phase-aware workload characterization framework for LLM inference on edge SoCs that enables reproducible, phase-aware characterization of edge LLM inference.
Amir Taherin, Sana Taghipour Anvari, Charles Amante et al.· 1 citation
This paper describes DeepSig's CUDA-based acceleration backend for the OCUDU physical layer and O-RAN fronthaul path, integrated through acceleration interfaces that are largely independent of the underlying acceleration mechanism. The design accelerates PDSCH, PUSCH, SRS, PRACH, split-8 lower-PHY transforms, and O-RAN fronthaul (O-FH) IQ compression/decompression while preserving existing factories, resource-grid interfaces, PRACH-buffer interfaces, and channel processors. CUDA-visible grids, device-side softbit buffers, stream events, pinned staging buffers, and managed-memory policies keep data resident on the accelerator when the platform and radio split permit it. On an NVIDIA DGX Spark platform with a GB10 GPU and ARM CPU host, representative measurements with CPU baselines pinned to high-capacity cores show up to 10.3x PUSCH speedup, 2.7x PDSCH speedup, 19.7x split-8 low-PHY RX speedup with slot-shaped batching and scattered mapped zero-copy, 91.4x O-FH BFP12 decompression speedup, and 28.8x PRACH detector speedup against the production CPU path, with CPU and GPU 10% BLER thresholds agreeing to within 0.064 dB in the tested PUSCH sweeps. The same resident pipeline provides an execution substrate for AI-RAN, allowing machine-learned channel estimation, neural receivers, and AI-native air-interface research to run beside standards-compliant baseband kernels.
M. Pennybacker, Wanze Liu, A. Kharchenko et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
