This work introduces Calculon-MoE, a tool that jointly explores FLOPS, HBM bandwidth and capacity, multiple network topologies, and popular parallelism/optimization strategies used in LLMs, and quantifies the benefits of overlapping compute and communication, hardware-accelerated collectives, widening the scale-up domain, and higher memory bandwidth and capacity.
Jesmin Jahan Tithi, Hanjiang Wu, Joongun Park et al.· International Conference on...· 0 citations
DynamoServe is presented, a multi-tenant LLM serving framework that addresses challenges through three key innovations: leveraging stranded GPU memory to offload model weights and KV caches, mitigating resource fragmentation in multi-workload environments, and improving memory locality through coordinated data placement and demand-driven weight migration across GPUs.
Diman Zad Tootaghaj, Khaled Diab, Bob Lantz et al.· Conference on Applications,...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.