Xu-Qi Zhu

SGLite: A Compact Serving Framework for Large Language Model Inference

Large language model (LLM) serving requires coordinated request admission, scheduling, KV-cache management, attention computation, CUDA Graph capture and replay, and distributed model execution. Established serving frameworks such as SGLang and vLLM deliver high throughput, but their large codebases make individual ser...