SimServing: Native-Execution Simulation for Evolution-Resilient LLM Serving Configuration Tuning

Abstract

Large Language Model (LLM) services expose a large configuration space that critically affects performance, yet tuning these configurations on hardware (e.g., NVIDIA GPUs) is prohibitively expensive. Simulation offers a scalable alternative, but state-of-the-art simulators suffer from a fundamental software engineering pitfall: they re-implement the serving framework's complex control logic (e.g., request scheduling and memory management), creating a semantic copy that inevitably diverges from the rapidly evolving upstream software. Our empirical study across eight consecutive vLLM versions confirms this fidelity drift, with prediction errors widening as the framework evolves. We observe that the serving management software dominates performance fidelity yet accounts for less than 10% of total execution time, with GPU computation consuming over 90%. This asymmetry motivates SimServing, a hybrid simulation framework that executes the native, unmodified serving software while replacing GPU computation with a pluggable latency estimator. This design preserves the serving management software's control flow by executing it rather than modeling it, thereby automatically inheriting upstream updates and substantially reducing the maintenance burden associated with re-implementation. Integrating SimServing with Particle Swarm Optimization for configuration tuning, our evaluation shows that it reduces throughput prediction error from 13.8% to 4.9%, achieves up to 21.18× simulation speedup, and discovers configurations that improve throughput by up to 10.2% over existing baselines.