Distributed large language model (LLM) systems rely on compute-communication (CC) overlap to hide communication overhead. However, scheduling compute and communication concurrently and maximizing overlap opportunities do not guarantee effective overlap. Resource contention can slow concurrent operations, while communic...
Jihwan Oh, Junkyum Kim, Seokjin Go et al.· IEEE International Symposium...· 0 citations
NELSSA is presented, an LLM serving system that integrates GPUs with real-world Processing-near-Memory accelerator devices to efficiently support mixed-length workloads and improves decode throughput by up to 5.5x in tokens/sec and reduces P99 latency by up to 15x compared to GPU-only baselines.
Sookyung Choi, Seungyong Lee, Kangkyu Park et al.· arXiv.org· 0 citations
Simthesizer introduces a composable simulator infrastructure that uniformly expresses the complete serving workflow, including the control decisions that coordinate it, and realizes it as a unified dynamic graph in Simthesizer simulator.
Wonung Kim, Hyunmin Choi, Minsu Kim et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.