TL;DR
This work investigates lightweight BF16 weight compression schemes tailored for PIM-based LLM inference by focusing on exponent-oriented compression methods that exploit the locality and redundancy present in BF16 exponent fields.
The rapid growth of Large Language Models has outpaced GPU memory capacity, making state-of-the-art models unfit for most GPUs. This necessitates evicting model parameters to CPU memory or disk, further away from the processing elements. In these scenarios, inference throughput is restricted by interconnect bandwidth, and while quantization schemes like FP8 reduce memory usage and enable larger models to fit smaller memories, they compromise generation quality and rely on specialized hardware support for efficient inference. To this end, we propose an inference pipeline that utilizes lossless entropy encoding (rANS) alleviating the bottleneck while maintaining standard floating-point portability or output precision. We demonstrate the potential of this approach by integrating the pipeline into vLLM, with a hardware-aware parallelization strategy. Our method increases the effective GPU memory capacity up to 30% in inference workloads with evicted weights, with no loss of quality and at the same time achieving speed up of 2.5–4.4%.
FLINT is proposed, a workload-driven HBF substrate for capacity-scalable LLM inference that integrates HBF as a memory-capacity tier alongside HBM while addressing three adoption challenges.
StreamDQ is proposed, a lightweight architectural enhancement that enables on-the-fly dequantization in the memory subsystem for high-throughput, large-batch LLM inference and reduces latency and improves decode throughput for end-to-end LLM inference.
SLIM (Saturation-Aware Lightweight Performance Model), a semi-analytical model that predicts LLM inference throughput and latency from analytical formulations of Transformer computation and memory traffic, is introduced, which outperforms representative performance-modeling baselines while successfully generalizing to previously unseen operating conditions.
Design rules and a reproducible evaluation protocol are contributed that jointly report quality, memory, and end-to-end speed, and a foundation for automated pipeline search under realistic single-GPU constraints is provided.
This paper presents FlashAttention-V, a blocked FlashAttention for scalable vector architectures that adapts efficiently from short to very long vectors by exploiting parallelism across attention heads, inter-head packing to enable efficient utilization of vector lengths beyond the head dimension, and improving vector register utilization and memory access locality.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
