WAKMan: Workload Aware KVCache Manager
Prefix caching enables a Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation stage for requests that share a prefix with earlier prompts, thus reducing latency and serving cost. With limited memory resource, the KV cache eviction policy becomes a decisive factor in d...
Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, S. Varghese et al.
· 0 citations