Duong The Do

Robust KV Cache Management for LLM Serving under Output Token Length Uncertainty

This work presents a robust KV cache management framework for LLM serving that jointly optimizes GPU parallelism configuration, KV cache reservation per request class, request routing across heterogeneous serving groups, and prefix caching for shared prompts that incorporates latency SLO constraints and captures the interaction between memory allocation, throughput, and queueing delay.