SparseEngine: Sparse-First Inference Engine
Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specif...
Jitai Hao, Quan-Sheng Gu, Qiang Huang et al.
· 0 citations