Disaggregated KV-Cache Fabric: Hardware-Agnostic RAM-Tiering, PCIe DMA Eviction, and Fine-Grained Prefix Deduplication for Autonomous Agent Swarms

Abstract

Modern Large Language Model (LLM) serving systems face an acute memory wall dominated by Key-Value (KV) cache allocation rather than model weight storage. In multi-agent autonomous swarms—where dozens of concurrent agents share extensive system prompts, tool schemas, and branching conversational trajectories—redundant KV cache allocations rapidly exhaust accelerator high-bandwidth memory (HBM), precipitating Out-Of-Memory (OOM) faults. We present KV-Cache Fabric, an open-source disaggregated architecture designed to investigate hierarchical memory tiering and concurrent prefix deduplication across three co-designed subsystems: 1. A tiered physical memory manager in Rust establishing LRU watermark eviction across host and device buffers with two-phase lock re-verification;2. A concurrent, fine-grained Radix Prefix Tree in Go utilizing hand-over-hand lock coupling down the tree and committing physical block identifiers at strict 16-token intervals; and3. A disaggregated routing conductor coordinating prefill workers and decode workers over gRPC. We evaluate the system under real-world multi-agent traffic profiles using live local model inference (SmolLM2-1.7B-Instruct) on client hardware. Empirical results across N=10 statistical trials demonstrate a 46.1% reduction in Time-To-First-Token (TTFT) on branching agent tasks, a 51.8% TTFT reduction on exact duplicates, a 12.4x collapse in prefill latency jitter (sigma = 138.59ms to 11.19ms), and steady-state decode throughput of 24.05 +- 1.10 tok/s. Source code, evaluation harness, and implementation artifacts are openly available at: https://github.com/kaunteyaarjun/kv-cache-fabric