Sheng Qi

Compositional Performance Modeling of LLM Inference Clusters with Hierarchical KV Caching

Design-space exploration of LLM inference clusters requires predicting how hardware and serving policies interact with hierarchical key-value (KV) caching. Isolated operation profiles alone do not determine cluster latency: reuse changes computational demand, while KV location, capacity, and contention govern execution...