TokenRouter: Efficient Serving System for Token-Level LLM Routing
Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing...
Tianyu Fu, Teng-Xuan Liu, Ruo-Xin Wang et al.
· 0 citations