When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
Long-running LLM applications repeatedly send growing context, making prefix caching critical for reducing prefill cost. Yet prefix-cache behavior under agentic workloads remains poorly understood. We study production traces from two companies and evaluate 14 eviction algorithms across HBM-constrained and large memory-...
Yi-Yu Liu, Min-Lan Yu, Jun-Cheng Yang
· 0 citations