Cutting to the Chase: NVMe-oF Fast-track for Replicated Persistent Key-value Stores

Abstract

Adopting replicated persistent key-value stores (RPKVSs) as metadata backends is becoming increasingly popular in modern large-scale storage systems. Conventional RPC-based KV replication is infamously subject to its replication overheads due to the inherent architectural mismatch between the application-level network stack and storage-prioritized operations. In this paper, we shift toward a new paradigm of storage-native block-based replication to fundamentally address the limitations of conventional replication. This shift is made possible by using widely deployed NVMe-oF technology to unify replication and storage efficiency under integrated block semantics. We present NoFDB, an NVMe-oF-enabled RPKVS design, that eliminates redundant index management and reduces replication latency, and introduces a set of system-level optimizations to address the challenges introduced by NVMe-oF and unlock its performance potential. Our evaluation shows that NoFDB improves random write throughput by up to 2.85 × and macrobenchmark throughput by up to 3.10 × over the best-performing baseline in each setting.