Book
Open access
Aug 2026
Towards Efficient Serving of Network-intensive LLM Inferences
Prefix caching has become a key technique for LLM serving, and nowadays the reusable KVCache contents are often hosted on distributed servers. For long-context LLM inferences with high cache hit ratio, cross-server KVCache transmission has become an emerging performance bottleneck; such network-intensive LLM inferences...
Weiye Wang, Chen Chen, Junxue Zhang et al.
· Asia-Pacific Workshop on Net... · 0 citations