Zero-Copy In-Place Compaction and Idempotent Associative Routing of Dynamic Key-Value Cache Tensors in Deep Learning Accelerators

Abstract

The serving of autoregressive Large Language Models (LLMs) is severely constrained by the computational and memory capacity limits of the Key-Value (KV) cache, commonly referred to as the Memory Wall. While dynamic context pruning algorithms mitigate memory expansion by discarding tokens of low attention mass, conventional systems incur severe hardware overheads: they either require out-of-place memory allocation spikes (O(N) auxiliary buffers via operating system calls like cudaMalloc) or introduce high page-table indirection latencies via virtualized paged attention mechanisms. In this paper, we propose a novel hardware-software co-designed architecture and GPU execution kernel for zero-copy in-place KV-cache compaction. By formulating eviction as an algebraic mapping satisfying the idempotence condition (f(f(x)) = f(x)), our method stabilizes retained tokens into mathematical fixed points and partitions the permutation space into mutually disjoint permutation cycles. We prove that minimal-index cycle leaders can be deterministically verified on-the-fly with strictly O(1) scalar auxiliary memory, completely eliminating marking bitmasks and temporary global buffers. We implement our algorithm as a high-performance Triton kernel and evaluate it on an NVIDIA Ada Lovelace GPU accelerator (RTX 500 Generation, sm_89) with a sequence length of 8,192 tokens and 50% context pruning. Empirical results demonstrate a 100% elimination of peak auxiliary VRAM (dropping from 384.00 MB to exactly 0.00 MB), zero numerical degradation, and guaranteed physical memory contiguity for downstream tensor cores.