GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving
GAUSS is presented, a workload-aware analytical and Monte-Carlo simulator that predicts the full distributions of time-to-first-token (TTFT) and time-between-tokens (TBT), together with the sustainable request rate, for a given workload, engine, and accelerator.
Srikant Bharadwaj, Ankur Mallick, S. Mastorakis et al.
· 0 citations