When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
This work studies production traces from two companies and evaluates 14 eviction algorithms across HBM-constrained and large memory-pool settings to show that effective prefix-cache management should retain recency as its foundation while selectively adding quick demotion for one-hit prefixes.