SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
SmartGen is designed, a KV cache transfer engine that allows seamless disaggregated LLM inference with three data transfer paths that reduces time-to-second-token by up to 4.3x compared with the typical full KV cache transfer approach while offering comparable subsequent decoding performance and accuracy.