SGLite: A Compact Serving Framework for Large Language Model Inference

Abstract

Large language model (LLM) serving requires coordinated request admission, scheduling, KV-cache management, attention computation, CUDA Graph capture and replay, and distributed model execution. Established serving frameworks such as SGLang and vLLM deliver high throughput, but their large codebases make individual serving mechanisms difficult to inspect, modify, and extend. This paper presents SGLite, a compact LLM serving runtime that preserves the core mechanisms of modern serving systems while keeping the serving path directly traceable. SGLite integrates FlashInfer-based attention, paged KV-cache management with Radix Cache prefix reuse, chunked prefill, prefill/decode overlap scheduling, CUDA Graph capture and replay, and tensor-parallel execution in a shared core supporting offline generation and online serving. Its implementation contains approximately 10,000 lines of core Python code with 23 runtime dependencies. We evaluate SGLite against SGLang and vLLM on Qwen3 models using NVIDIA H200 and NVIDIA GeForce RTX 5090 GPUs. Across offline and online settings, SGLite approaches the throughput of established serving frameworks while maintaining a substantially smaller and more transparent implementation. These results indicate that practical LLM serving performance can be achieved in a compact codebase, making SGLite suitable for experimentation, debugging, education, and systems research.