Warming up Storage-Level Caches with Bonfire

Gutenberg: Taming Latency-Critical Cloud Services with Near-Data-Processing

Gutenberg, a CPU+NDP for mutable, latency-critical cloud services, outperforms prior systems, reducing average and p99 latency by up to 80.4% and 85.8% and improves isolation and fairness while adapting to changing workload behaviors.

NyxCache: Flexible and Efficient Multi-tenant Persistent Memory Caching

The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems

Prefix caching is critical for efficient large language model (LLM) serving, particularly for agentic workloads that repeatedly invoke the model with a growing conversation and tool-use history. By reusing the key-value (KV) states of previously processed prefixes, prefix caching avoids redundant prefill computation. I...

ByPASS: Selective Cache Bypassing of Host Writes in Computational Solid-state Drives

Computational solid-state drives (CompSSDs) are evolving into intelligent storage solutions that integrate near-data computing architectures to achieve computation-storage convergence. They can reduce the overhead of data migration and then accelerate the execution of data-centric workloads through in-situ processing....

LLM KV-cache: To Restore or To Recompute, That Is the Question

The challenges of the restoration-recomputation trade-off are investigated and its impact on inference performance when left unaddressed, and an I/O-aware KV-cache management policy is presented that dynamically navigates this trade-off.

Buoy: Efficient and Effective Cache Replacement for Prefix Caching

Modern large language model (LLM) systems widely employ prefix caching to enable key-value (KV) cache reuse across different queries to minimize inference costs. At the heart of prefix caching is the replacement algorithm, which is crucial for managing limited cache space across the GPU–CPU memory hierarchy. However, t...