Neuro-Elastic, an adaptive inference framework that operates at two granularities: per-input sparsity (entropy-driven early exit and token pruning) and device-state-driven switching among pre-compiled mixed-precision model variants, is proposed.
Abstract
Deploying compact Transformer encoders on resource-constrained edge devices forces a conflict between reasoning quality and tight latency, thermal, and energy budgets, yet static compression ignores both the variance in input difficulty and the changing state of the hardware. We propose Neuro-Elastic, an adaptive inference framework that operates at two granularities: per-input sparsity (entropy-driven early exit and token pruning) and device-state-driven switching among pre-compiled mixed-precision model variants. Its central contribution is this hardware and software co-design, not the individual mechanisms, which are established. Three components realize it: Hardware-Aware Differentiable Mixed-Precision Quantization (HA-DMPQ), which learns layer-wise bit-widths by gradient descent within the target accelerator’s native instruction set; LoRAQuant, a split-precision low-rank strategy for on-device fine-tuning; and Dynamic Sparsity Induction (DSI), which cuts memory bandwidth through entropy-driven exits and cache-aware token pruning. We measure all latency and energy on an NVIDIA Jetson Orin Nano and use a Qualcomm Hexagon 680 simulator only for instruction-level validation, scoping our claims accordingly. On the MASSIVE intent-classification benchmark with an mBERT-base backbone, the Eco operating point cuts latency by 46.2% and energy by 54.5% at a 1.6 percentage-point accuracy cost (95.2% versus the 96.8% FP16 teacher), while the Balanced operating point stays within 0.3 points of the teacher (96.5%) and still cuts latency by 31.9%. All results are reported as mean ± standard deviation over five independent runs.
The proposed hybrid pipeline includes structured pruning, INT8 quantization and task-specific knowledge distillation, which is benchmarked against standalone methods and reinforces the idea of upper bound projection based approach for accuracy-oriented, multi-level compression.
Krishna Kumar Tiwari, Komal Tahiliani, Uma Shankar Birthare et al.· International journal of com...· 0 citations
On-device model adaptation is essential to enable lifelong personalization on resource-constrained hardware, but compute, power, and memory limitations of such devices make end-to-end backpropagation impractical for modern deep neural networks. This work proposes a heterogeneous adaptation pipeline that repurposes a commercial edge AI inference accelerator, Hailo-8L, for frozen-backbone feature extraction during on-device training. The computational graph is partitioned so that the pre-trained backbone is quantized to INT8 and run on the accelerator, while only a lightweight FP32 classification head is fine-tuned on the host CPU, enabling frequent, energy-efficient in-field updates with most weights remaining fixed. Across multiple architectures and datasets, this pipeline achieves up to 15.4x faster wall-clock training time compared to a Raspberry Pi 5 CPU baseline, offers competitive throughput in favorable settings, and consistently reduces energy per sample. Post-training quantization restoration is shown to be crucial for preserving the quality of accelerator-generated features and mitigating accuracy loss in quantization-sensitive architectures. Overall, the results demonstrate a practical approach to efficient on-device adaptation using inference-oriented edge accelerators. The implementation is available at https://github.com/MatPiech/accelerator-training.
M. Piechocki, Alessandro Capotondi, Marek Kraft· arXiv.org· 0 citations
FluxBin is proposed, an algorithm-kernel co-design that synergizes post-training quantization with a highly optimized CUDA kernel and introduces Decoupled Row-Column Binary Decomposition to enhance representational capacity while maintaining hardware efficiency.
Qingyao Yang, Runming Yang, He Xiao et al.· 0 citations
APT, a software-hardware co-designed accelerator for high-resolution DiTs that leverages attention probabilities as a unified importance metric to jointly optimize computation through fine-grained pruning and adaptive precision scaling, and is evaluated on SOTA DiT models including PixArt, Stable Diffusion 3, and FLUX.
Sungyeob Yoo, Seeyeon Kim, Joonyong Park et al.· 0 citations
Findings confirm that combining complementary compression strategies yields substantially better performance-efficiency trade-offs than any single technique applied in isolation.
Upma Sharma Archana· International Journal of Res...· 0 citations
Deploying large-scale transformer models on resource-constrained edge devices remains a challenge due to the high energy and memory overhead inherent in static inference, which processes simple and complex tokens with uniform intensity. To address this, we propose Adaptive Model Compression (AMC), a saliency-driven framework that dynamically allocates hardware resources based on token importance. By implementing a multi-tier architecture, our system identifies critical high-saliency information for full-precision processing while aggressively reducing the rank and bit-width of less significant data. Experimental results demonstrate that AMC achieves a 59.2% reduction in system energy and a 2.24x increase in throughput on 45nm CMOS hardware. This approach effectively extends the battery life of mobile devices by utilizing high-definition compute only where necessary, maintaining robust performance with a marginal 3.6% accuracy trade-off.
Jiayin Hu, Kai Yuan, Vanessa Hu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
