Aug 2026· International Symposium on Low Power Electronics and Design· pp. 1-7· 0 citations· 31 references
Computer Science
TL;DR
H3-Attn is proposed, an Attention-efficient 3D DRAM PNM processor for low-batch LLM inference that features a hybrid head parallelism for Attention processing, whereby various optimized Attention mechanisms with spatial tiled FlashAttention can be flexibly enabled with fully leveraged 3D DRAM PNM bandwidth.
Abstract
Edge-side LLM deployment, characterized by low-batch inference, has gained significance due to data privacy and personalization requirements. Attention is the key operator of LLM inference, involving massive KV cache access and full-context-aware softmax computation. To optimize KV cache size and softmax computation, various Attention mechanisms and the FlashAttention algorithm have been proposed. As 3D DRAM-based process-near-memory (PNM) presents a promising solution for low-batch LLM inference, processing these optimized Attention mechanisms may suffer from low utilization of 3D DRAM internal bandwidth due to the distributed memory access nature. Furthermore, existing Special Function Unit (SFU) architectures in 3D DRAM PNM for FlashAttention suffer from either severe communication bottlenecks in centralized designs or stringent area constraints in distributed implementations. To address these challenges, we propose H3-Attn, an Attention-efficient 3D DRAM PNM processor for low-batch LLM inference. H3-Attn features a hybrid head parallelism for Attention processing, whereby various optimized Attention mechanisms with spatial tiled FlashAttention can be flexibly enabled with fully leveraged 3D DRAM PNM bandwidth. A Hierarchical SFU (H-SFU) architecture with Multi-Head Instruction Interleaving (MHII) is also proposed, which significantly enhances softmax efficiency in FlashAttention by co-optimizing SFU area footprint and temporal utilization. Evaluations on various LLM models demonstrate that our proposed hybrid head parallelism achieves a 1.54-3.84× decoding speedup over the baseline Attention processing. The H-SFU with MHII delivers a 3.20× compute density improvement compared with prior SFU architectures for 3D DRAM PNM.
FLINT is proposed, a workload-driven HBF substrate for capacity-scalable LLM inference that integrates HBF as a memory-capacity tier alongside HBM while addressing three adoption challenges.
Geraldo F. Oliveira, Arash Tavakkol, Xiang-Yu Zhu et al.· 0 citations
The emergence of Vision-Language Models (VLMs) has enabled multimodal reasoning, e.g. video understanding, yet their extension to long-context inference remains bottlenecked by the “token explosion”. This surge in sequence length leads to prohibitive attention computation overhead and memory-bound KV cache access. While 3D-stacked logic-to-DRAM architectures offer high-bandwidth Processing Near Memory (PNM) capability, their distributed memory banks face severe workload imbalance due to the unique spatiotemporal sparsity patterns in video understanding tasks. In this paper, we introduce MVP, a 3D-stacked VLM accelerator featuring context-aware sparse attention (CASA) and online workload-aware hybrid parallelism scheduling. By leveraging dynamic sparse attention patterns, our design prunes redundant attention computation FLOPS and adaptively balances computation across hybrid bonding (HB) based many-core NoC architecture. Experimental results on VQA tasks demonstrate that our architecture achieves a 5.97× speedup and a 5.42× energy-efficiency improvement over the RTX 4080 GPU deployment, effectively mitigating the bottlenecks of VLM inference for video-understanding on mobile devices.
Yifan Ding, Qianxu Wang, Dunshan Yu et al.· International Symposium on L...· 0 citations
This paper presents a heterogeneous decode-phase serving system that relocates the KV cache out of GPU memory, motivated by the retrieval-based sparse attention that recent frontier LLMs adopt to serve million-token contexts. It partitions a decode step by operation type: GPU nodes hold the model weights and execute the projections and MoE layers, while processing-near-memory (PNM) nodes hold the KV cache and index keys and execute every operation that reads them. We first show that the assumptions behind prior PIM and PNM designs no longer hold for these operations, and derive four design requirements for such a node. From these requirements, we propose KARAT (KV-cache-resident Accelerator for Retrieval-based ATtention), a general-purpose PNM design that is the design point meeting all four. A KARAT device combines large LPDDR capacity with general-purpose compute sized for the retrieval indexer, serving an operational intensity beyond what PIM/PNM designs built for low-intensity GEMV target while accommodating diverse sparse attention algorithms that fixed-function units cannot support as they evolve. To reduce pipeline bubbles as the two device types alternate between micro-batches, we further propose opportunistic, fine-grained micro-batch scheduling (OFMS), which hides expert all-to-all behind the other micro-batch's GEMMs, and context-length-aware micro-batch rebalancing (CMR), which equalizes their token counts despite the variance in context length. Across three state-of-the-art models and real agentic traces, our proposed system improves throughput per TDP under a service-level objective by 2.09-6.13x over a GPU-only baseline and runs training-free sparse attention methods with 1.36-3.21x improvements.
Hyungkyu Ham, Junhyeong Bae, Seungheon Lee et al.· 1 citation
OasisKV is presented, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding and observes that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD).
LLM inference has become an essential service, yet it imposes unprecedented demands on memory bandwidth, computational density, and communication efficiency. While IMC is a promising solution to the memory wall issue, the heterogeneous data dynamicity of LLM requires complementary resources to handle intermediate data generated during run-time. Furthermore, the massive number of parameters in LLM necessitates scale-up architectures where on-chip data movement is often the primary performance bottleneck. This article presents a hardware-software co-design framework that unifies distributed compute, memory, and communication into a seamless processing-communication fabric. On the hardware side, we propose a scalable architecture, named LEAP, that integrates IMC PE, NMC PE, and INC. This allows each hardware layer to execute specialized tasks: IMC for static weights, NMC for dynamic data, and INC for partial result reduction. On the software side, we introduce a partitioning, mapping, and scheduling framework optimized for key metrics in LLM serving, including throughput and latency. To address the distinct computational intensities of the prefill and decode phases, we present a prefill-decode disaggregation approach that dynamically reconfigures PE organizations to maximize resource utilization. Compared to commercial GPU platforms, the proposed architecture provides a throughput and an energy efficiency improvement of $\geq{}1.52\times$ and $24.91\times$, respectively.
This paper presents FlashAttention-V, a blocked FlashAttention for scalable vector architectures that adapts efficiently from short to very long vectors by exploiting parallelism across attention heads, inter-head packing to enable efficient utilization of vector lengths beyond the head dimension, and improving vector register utilization and memory access locality.
