PMKS: Co-Designing Distributed Networking and Multi-Tier Storage for Ultra-Long Context LLM Inference

Abstract

The massive memory footprint of the KV Cache is a critical bottleneck in scaling Large Language Model (LLM) inference to ultra-long contexts. Existing single-node or compression-based methods either compromise model fidelity or fail to scale efficiently. This paper introduces a four-tier heterogeneous storage architecture that extends the KV Cache across GPU VRAM (L1), host RAM (L2), local NVMe SSDs (L3), and an RDMA-enabled distributed fabric (L4). To mask cross-tier I/O latency, we propose the Predictive Multi-tier KV Scheduling (PMKS) algorithm, which exploits the sequential access patterns of autoregressive decoding to prefetch blocks asynchronously. We evaluate the system on an ODCC AI Storage Lab testbed using DeepSeek-R1 with context lengths up to 100K tokens. Compared to a vanilla vLLM baseline under batch-size-16 workloads, our architecture achieves a 90% reduction in Time-to-First-Token (TTFT) for warm-start scenarios and a 20-fold increase in token throughput, while maintaining stable decoding latency. Finally, a TCO analysis indicates that storage-augmented GDDR-based GPU nodes can deliver long-context throughput comparable to premium HBM-based infrastructures at a lower system cost, offering a cost-effective paradigm for enterprise-scale LLM deployment.