SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independent requests favor data-parallel attention, and long prompts favor context parallelism. Reasoning, agentic, and RL-rollout workloads make a fixed choice untenable: a batch...
Chuan Liu, Shuo-Ming Zhang, Zhi-Cheng Li et al.
· 0 citations