Jul 2026· IEEE International Conference on Cloud Computing· pp. 290-296· 0 citations· 30 references
Abstract
The rapid growth of Large Language Models has outpaced GPU memory capacity, making state-of-the-art models unfit for most GPUs. This necessitates evicting model parameters to CPU memory or disk, further away from the processing elements. In these scenarios, inference throughput is restricted by interconnect bandwidth, and while quantization schemes like FP8 reduce memory usage and enable larger models to fit smaller memories, they compromise generation quality and rely on specialized hardware support for efficient inference. To this end, we propose an inference pipeline that utilizes lossless entropy encoding (rANS) alleviating the bottleneck while maintaining standard floating-point portability or output precision. We demonstrate the potential of this approach by integrating the pipeline into vLLM, with a hardware-aware parallelization strategy. Our method increases the effective GPU memory capacity up to 30% in inference workloads with evicted weights, with no loss of quality and at the same time achieving speed up of 2.5–4.4%.
Recent advances in transformer-based foundation models have made them the default choice for many tasks, but their rapidly growing size makes fitting a full model on a single GPU increasingly difficult and their computational cost prohibitive. Block low-rank (BLR) compression techniques address this challenge by learning compact representations of weight matrices. While traditional low-rank (LR) methods often incur sharp accuracy drops, BLR approaches such as Monarch and BLAST can better capture the underlying structure, thus preserving accuracy while reducing computations and memory footprints. In this work, we use roofline analysis to show that, although BLR methods achieve theoretical savings and practical speedups for single-token inference, multi-token inference often becomes memory-bound in practice, increasing latency despite compiler-level optimizations in PyTorch. To address this, we introduce custom Triton kernels with partial fusion and memory layout optimizations for both Monarch and BLAST. On memory-constrained NVIDIA GPUs such as Jetson Orin Nano and A40, our kernels deliver up to 3.76 × speedups and 3 × model size compression over PyTorch dense baselines using CUDA backend and compiler-level optimizations, while supporting various models including Llama-7/1B, GPT2-S, DiT-XL/2, and ViT-B.
Pierre Abillama, Changwoo Lee, Juechu Dong et al.· IEEE International Symposium...· 0 citations
Beyond is presented, a drop-in runtime that integrates a smart offloading scheme to selectively identify and retain salient KV entries across continuous decoding sessions, together with a hybrid CPU–GPU attention mechanism for scalable inference.
Weishu Deng, Yujie Yang, Peiran Du et al.· IEEE International Symposium...· 1 citation
This study presents a cross-platform, multi-model empirical study, where several important observations are brought, including the contrastive effect of quantization under different hardware bottlenecks, along with a quantification of runtime delays caused by the lack of parallelism in the ARM architecture.
Subhransu Das, Jiaming Cheng, Swathi Vallabhajosyula et al.· Practice and Experience in A...· 0 citations
With the widespread adoption of large language models (LLMs), the demand for graphics processing units (GPUs)-essential for accelerating LLM training and inference- has increased significantly. This has led to rising procurement and operation costs, imposing a significant financial burden on research institutions and industry. Efficient utilization of GPU resources has thus become a critical challenge. In this study, we investigate strategies to maximize resource efficiency by hosting multiple models on a single GPU rather than dedicating each GPU to a single model. We examined various GPU resource partitioning approaches to improve the utilization of limited GPU resources. Specifically, we compared two resource allocation methods for concurrently executing two models on a single GPU: using vLLM, a high-performance LLM inference framework, and using NVIDIA Multi-Instance GPU (MIG). The results demonstrated that the MIG configuration increased total throughput by approximately 700-950 tokens/s compared with vLLM-only execution, suggesting that partitioning a GPU into independent MIG instances can improve throughput for concurrent model execution.
Madoka Takato, Akira Hirai, Reika Kinoshita et al.· Annual International Compute...· 0 citations
This work evaluates a CXL memory expander equipped with an on-device DMA controller as a high-performance staging tier in GDS-style data paths, sustaining high bandwidth for fine-grained LLM inference workloads such as KV cache updates.
Adaptive Sequence Pipeline Parallel Offloading (SPPO) is proposed, a novel framework that optimizes memory and computational resource efficiency for long-sequence LLM training and develops an adaptive pipeline scheduling approach with a heuristic solver and multiplexed sequence partitioning to improve computational resource efficiency.
Qiaoling Chen, Shenggui Li, Wei Gao et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
