Jul 2026· International Conference on Supercomputing· pp. 286-300· 0 citations· 46 references
Computer Science
TL;DR
CoCache is presented, a cooperative cache-management framework that continuously tunes the metadata-data cache split and combines lightweight online hotness tracking with a unified latency model that captures the coupled impact of metadata and data caching on the read path.
Abstract
Modern LSM-based KV stores reduce read amplification with two in-memory caches, namely a table cache for metadata such as index and Bloom-filter blocks, and a data cache for value blocks. These caches draw from a shared memory budget and are jointly exercised on the read path, so the metadata-data split is inherently coupled and end-to-end read latency can be non-monotonic in the allocation. Allocating more memory to one cache may improve its hit rate but evict blocks from the other, yielding hard-to-predict performance especially under dynamic workloads. Most systems therefore rely on fixed, ratio-based heuristics, which can be far from optimal. We present CoCache, a cooperative cache-management framework that continuously tunes the metadata-data cache split. CoCache combines lightweight online hotness tracking with a unified latency model that captures the coupled impact of metadata and data caching on the read path. Using these signals, CoCache efficiently searches candidate splits and applies the one with the lowest predicted latency, then re-optimizes as access patterns shift. We implement CoCache in RocksDB and evaluate it on synthetic, benchmark, and production-derived workloads. Compared to state-of-the-art baselines, CoCache improves cache hit rates by up to 1.56 × , increases read throughput by up to 1.43 × , and reduces read latency by up to 31.2%.
Log-structured merge-tree (LSM-tree) key-value stores rely on caching to mitigate multi-component lookups and long tails, yet block and KP caches are prone to compactioninduced expiry, and all three cache types suffer from scan pollution under LRU eviction. We present AutoThermKV, a two-level in-memory architecture that decouples fast admission from hot-item protection. The front tier (HotFilter) uses a 64 Baligned bitmap/key/value page layout with Quick Space Reservation for near-constant admission cost; the back tier (HotCommitted) retains proven hot items via a lightweight LRU and a secondtouch promotion rule. Implemented atop RocksDB and evaluated on YCSB-like workloads with controllable temporal locality, AutoThermKV achieves up to 2.58 × throughput speedup, 2.60 × average-latency reduction, and tail-latency improvements of up to 20 × (p99) and 36 (p999) under modest memory budgets.
Yunfan Chi, E. Sha, Longshan Xu et al.· IEEE International Conferenc...· 0 citations
This model reveals one key opportunity: dividing a restore request proportionally between the storage path and the GPU can improve inference performance while still meeting SLOs, and reduces the KV-cache storage stack to a performance model based on per-tier capacity, per-tier and interconnect bandwidth, and GPU arithmetic throughput to identify optimization opportunities for KV-cache management.
Read latency is a critical bottleneck for NAND-based SSDs in AI-driven datacenter workloads, where model parameters and key-value data are frequently swapped between main memory and storage. Existing prefetching schemes operate on block-level address sequences that have been stripped of application context by the filesystem layer, limiting their prediction accuracy and coverage. We propose the BASIC Prefetcher, which recovers application-level I/O patterns by classifying requests into size-based bins and modeling temporal bin transitions with a lightweight Markov-chain framework. By confining address prediction to frequently accessed bins and their primary data regions, BASIC Prefetcher achieves up to 89% prediction accuracy on sequential workloads and 70% on highly random workloads, including AI large language model inference traces, without requiring pre-training or per-page state tracking. Across six diverse workloads, our approach improves SSD throughput by 1.41× to 2.39× over existing prefetching schemes.
Han Jang, Dongjun Lee, Youngbin Jin et al.· Proceedings of the 19th ACM...· 0 citations
Flash memory, characterized by its rapid read and write capabilities and compact size compared to traditional hard disk drives (HDDs), has become a dominant choice for modern storage devices, particularly solid-state drives (SSDs). Despite its advantages, flash memory suffers from device wear caused by repeated write and erase operations, which increases the probability of read errors and ultimately limits SSD lifespan. To enhance SSD reliability and performance without incurring significant resource overhead, prior research has explored data allocation strategies that place frequently accessed data into lower-latency flash locations. However, such schemes typically rely on per-page access counters or LRU-based buffer management, both of which introduce non-negligible metadata maintenance overhead on resource-constrained SSD controllers. This paper introduces a Lightweight Cache Buffer Management (LCBM) strategy that avoids precise frequency tracking altogether. LCBM combines a batch-based first-in-first-out (FIFO) buffer policy with a simple swapping mechanism that dynamically adjusts data placement across the least significant bit (LSB), central significant bit (CSB), and most significant bit (MSB) positions, keeping frequently accessed data in cells with shorter read latency while fully exploiting multi-plane and intra-channel parallelism. Trace-driven simulation shows that, relative to the state-of-the-art FSPDA, LCBM reduces the average read latency by 13.62% and the write response time by 66.81% (and by 16.82% and 64.97%, respectively, over OSPADA), while eliminating the metadata overhead of counter- and LRU-based designs. We further validate LCBM at the system level on the FEMU emulator using both Filebench workloads and real-user traces, where it sustains competitive-to-higher throughput under realistic, long-term deployment scenarios.
OasisKV is presented, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding and observes that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD).
Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache. We characterize this tradeoff in vLLM across GPU, CPU, and NVMe tiers using synthetic workloads, long-context benchmarks, production traces, and find that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth. These findings motivate py-kvcache, a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading, which starts disk reads while requests are still waiting, overlapping with compute. At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x. With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation. LongBench and SCBench show that these benefits extend to irregular prefix chains and multi-turn workloads. Bailian trace replays improve TTFT on a weaker GPU, but on an H100 the average request falls below the break-even point and GPU memory alone retains enough prefixes. External KV caching should therefore be treated as a setup specific admission decision. The py-kvcacheimplementation is available at: https://github.com/atlarge-research/py-kvcache.
Joseph Kanichai, T. De Matteis, Animesh Trivedi· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
