Replacing NVMe Staging in LLM Inference with a High-Bandwidth CXL Memory Expander with an On-Device DMA Controller

Abstract

The rapid growth of LLM inference has exposed critical bottlenecks in GPU memory capacity and data movement efficiency. When GPU memory is exhausted, modern inference frameworks rely on GPUDirect Storage (GDS) with PCIe P2P transfers to stage data—such as KV caches and intermediate tensors—between GPU and NVMe-based storage. NVMe storage remains a significant performance bottleneck, with state-of-the-art enterprise PCIe Gen5 SSDs delivering only 14 GB/s sequential reads and 7 GB/s writes per drive. In this work, we evaluate a CXL memory expander equipped with an on-device DMA controller as a high-performance staging tier in GDS-style data paths. Although contemporary GPUs do not natively participate in the CXL protocol, our design works around this gap: the device-side DMA controller drives PCIe peer-to-peer (P2P) transfers directly against GPU HBM, while CXL is used only on the host side for capacity expansion and device management. We develop a custom NIXL backend plugin to enable benchmarking of GPU-to-expander data movement, and use NIXLBench to characterize transfer performance across transfer sizes and request parallelism configurations. The expander achieves 51 GB/s read bandwidth (~80% of the PCIe Gen5 ×16 peak) and 33 GB/s write bandwidth—utilizing a substantially larger fraction of the PCIe Gen5 ×16 interconnect than leading Gen5 NVMe SSDs achieve on the same interconnect generation. The remaining ~1.5× read/write gap is consistent with PCIe transaction completion semantics and leaves room for further write-side optimization. Multi-threaded descriptor submission, a routine capability of modern inference frameworks, additionally lowers the block size at which peak bandwidth is reached from 8 MB to 256 KB, sustaining high bandwidth for fine-grained LLM inference workloads such as KV cache updates.