LODAR: layered optimization for distributed adaptive LLM routing
Large language model (LLM) serving platforms increasingly route requests across heterogeneous backends, transforming query routing from a model-selection task into a system-level load-balancing problem. Centralized routers can become throughput bottlenecks and single points of failure as deployment scale and request vo...