Rong-Le Wang

Adaptive Context Parallelism for Production LLM Serving

Vertumnus is presented, an adaptive CP serving system designed for heterogeneous and evolving workloads that reduces mean TTFT and improves token-weighted SLO attainment and introduces a global prefix-cache management policy that coordinates cache placement and replication among workers with the same or different CP de...