Jul 2026· International Conference on Wireless Communication Technology and Intelligent Signal Processing· Vol 14274, pp. 142740K - 142740K-5· 0 citations· 14 references
Engineering
TL;DR
The paper outlines an efficient scheme of inference in LLM by synergistically using post-training quantization, key-value cache compression, speculative decoding, and Flash Attention to fill the gap between the state-of-the-art LLM capabilities and the latency constraints of the intelligent signal processing systems.
Abstract
The impressive performance of large language models (LLMs) in various artificial intelligence tasks has not yet been applied to real-time signal processing tasks due to the prohibitive memory and computation requirements. The paper outlines an efficient scheme of inference in LLM by synergistically using post-training quantization, key-value (KV) cache compression, speculative decoding, and Flash Attention. We look at the combined effect that these techniques have on reducing memory footprint by as much as 4x and enhancing decoding by up to 3x without drastically compromising the accuracy of signal interpretation. The results of experimental assessments on spectrum sensing, modulation classification, and channel estimation tasks indicate that our optimized pipeline can reach almost the baseline performance, and it can be implemented on the resource-constrained hardware accelerators, such as GPUs, TPUs, and edge neural processing units. The suggested methodology will fill the gap between the state-of-the-art LLM capabilities and the latency constraints of the intelligent signal processing systems, paving a viable way to LLM-based cognition in the next-generation wireless and IoT networks.
This paper presents FastTPS, a high performance and low-precision loss method for accelerating the token-phase in LLM inference on general AI accelerators which includes three key components: AI accelerator-enabled reloading-free KV Cache concatenation which decreases memory access overhead as well as enables full fusion of Attention.
Wenzong Yang, Danyang Zhang, Kunteng Cao et al.· arXiv.org· 0 citations
The rapid growth of Large Language Models has outpaced GPU memory capacity, making state-of-the-art models unfit for most GPUs. This necessitates evicting model parameters to CPU memory or disk, further away from the processing elements. In these scenarios, inference throughput is restricted by interconnect bandwidth, and while quantization schemes like FP8 reduce memory usage and enable larger models to fit smaller memories, they compromise generation quality and rely on specialized hardware support for efficient inference. To this end, we propose an inference pipeline that utilizes lossless entropy encoding (rANS) alleviating the bottleneck while maintaining standard floating-point portability or output precision. We demonstrate the potential of this approach by integrating the pipeline into vLLM, with a hardware-aware parallelization strategy. Our method increases the effective GPU memory capacity up to 30% in inference workloads with evicted weights, with no loss of quality and at the same time achieving speed up of 2.5–4.4%.
Väinö-Waltteri Granat, Jakub Žádník, M. Mäkitalo et al.· IEEE International Conferenc...· 0 citations
This study presents a cross-platform, multi-model empirical study, where several important observations are brought, including the contrastive effect of quantization under different hardware bottlenecks, along with a quantification of runtime delays caused by the lack of parallelism in the ARM architecture.
Subhransu Das, Jiaming Cheng, Swathi Vallabhajosyula et al.· Practice and Experience in A...· 0 citations
Findings confirm that combining complementary compression strategies yields substantially better performance-efficiency trade-offs than any single technique applied in isolation.
Upma Sharma Archana· International Journal of Res...· 0 citations
Experimental results demonstrate that effective compression significantly reduces model size and computational cost with minimal performance loss, highlighting the importance of compression-aware design and concluding as a valuable reference for building efficient and scalable AI systems.
Daniel Rodríguez· International Journal of App...· 0 citations
The rapid development of programmable network devices and the widespread adoption of machine learning (ML) in networking have facilitated efficient research into intelligent data planes (IDPs). Offloading ML to programmable data planes (PDPs) enables quick analysis and responses to network traffic dynamics, and efficient management of network links. Compared to using an external low-cost board with sufficient memory and a general-purpose CPU, IDP deployment keeps inference inside the switch forwarding pipeline, avoiding inter-device transfer and coordination overhead. This enables line-rate processing and faster response for real-time network control. However, the hardware pipeline presents significant resource limitations. For instance, Intel Tofino ASIC has only 10Mb SRAM in each stage, and lacks support for multiplication, division, and floating-point operations. These constraints significantly hinder the development of IDP. This paper presents Quark, a framework that automatically compresses the convolutional neural network (CNN) and fully offloads quantized inference onto PDP. Quark employs model pruning to simplify the CNN model, uses quantization to support floating-point operations, and utilizes neural architecture search to balance accuracy and PDP resource constraints. Additionally, Quark divides the CNN into smaller units to improve resource utilization on the PDP. We have implemented a testbed prototype of Quark on both P4 hardware switch (Intel Tofino ASIC) and software switch (i.e., BMv2). Extensive evaluation results on the ISCX Botnet dataset demonstrate that Quark achieves 97.3% accuracy while using only 24.27% of the SRAM resources on the Intel Tofino ASIC switch, completing inference tasks at line rate with an average latency of $42.66\mu s$ .
Xiaoquan Zhang, Mai Zhang, Lin Cui et al.· IEEE Transactions on Network...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
