Empirical results on RouterBench and SWE-Bench demonstrate that WR-Offline surpasses existing baselines in performance under a fixed budget and adheres more closely to budget constraints, and WR-Online achieves comparable performance to the baselines, while using substantially less exploration data.
Abstract
Large language models (LLMs) achieve impressive performance across multiple domains, but using the most capable model for every query is prohibitive at scale. LLM routing exploits diversity in model capability and cost by assigning each query to a suitable model to balance utility and budget. Current methods have two limitations: (i) they either use heuristics that do not always enforce the budget constraint or impose a fixed per-query budget that cannot adapt across the workload and leads to suboptimal performance; (ii) they require supervised learning on a dense dataset with statistics for every query-model pair, which is expensive to collect. To address these challenges, we formulate LLM routing as a constrained contextual multi-armed bandit problem and introduce WISERouter (WR for short), a framework that supports offline learning from historical interactions as well as online learning with exploration. We further prove that WR-Online achieves a sublinear regret bound of $O(\sqrt{T})$ over a time horizon $T$. Empirical results on RouterBench and SWE-Bench demonstrate that (i) WR-Offline surpasses existing baselines in performance under a fixed budget and adheres more closely to budget constraints, and (ii) WR-Online achieves comparable performance to the baselines, while using substantially less exploration data.
Routing among large language models (LLMs) trades response quality against serving cost, motivated by the reported gap between deployed routers and a per-instance oracle. Recent analysis shows that test-time resampling can recover per-instance selection headroom that no single-commit router captures; however, that guarantee holds only under an idealized oracle equipped with correctness labels and an unconstrained budget, neither of which a deployed system has. To the best of our knowledge, no previous work treats resampling the committed model and rerouting to an alternative model as competing uses of a single per-query cost budget. Therefore, this work formulates budget-aware test-time model selection: given a per-query budget and an imperfect verifier, allocate each unit of budget between resampling and rerouting so that expected correctness is maximized. An online resample-or-reroute (RoR) allocation policy driven by estimated marginal correctness per unit cost is proposed, and its behavior is grounded in the recoverability asymmetry between selection and sampling. Replay experiments on newly regenerated multi-draw correctness tensors from an eleven-model open-weight pool over four benchmarks of differing difficulty show that the proposed RoR policy attains a favorable cost-quality Pareto front relative to single-route, one-commit-router, budget-aware best-of-K, cascade, and random-allocation baselines for the tested pools, with the largest gains on the most heterogeneous benchmark; an ablation further shows the gains are verifier-gated, shrinking as verifier quality degrades, and robustness replays under a provider price vector and a label-free agreement verifier delineate where the conclusions carry over.
The rapid proliferation of Large Language Models (LLMs) with varying capability profiles, context window limits, execution latencies, and financial costs presents a significant operational challenge for enterprise AI deployments. Monolithic deployment strategies wherein all requests are directed to a single high-capability frontier model result in substantial compute over-provisioning and excessive operational costs for routine queries. Conversely, relying solely on lightweight models degrades output accuracy on complex multi-step reasoning tasks. To resolve this trade-off, this paper introduces LLM-Advisor, an open-source, adaptive framework designed for intelligent query categorization, dynamic model evaluation, and constraint-aware request routing across heterogeneous multi-LLM pools. LLM-Advisor analyzes incoming prompt features, structural complexity, domain requirements, and user-defined constraints (e.g., maximum cost per request, latency thresholds) to route tasks to the optimal candidate model. We evaluate LLM-Advisor using a benchmark suite of 1,000 queries across code generation, general reasoning, and contextual retrieval tasks using both proprietary and open-weight models (including GPT-4o, Claude 3.5 Sonnet, Llama 3, and Mistral). Experimental results demonstrate that LLM-Advisor achieves a 42% reduction in overall inference expenditure and a 35% decrease in average response latency while retaining 94.6% task accuracy compared to static GPT-4o baseline routing. These findings highlight LLM-Advisor as an efficient, highly scalable middleware solution for production-grade AI system deployments.
Harshil Lodhiya· International Journal of Res...· 0 citations
Large language models are increasingly used in practical systems, making efficient model selection important for reducing deployment cost. LLM routing has emerged as a practical solution for allocating each input query to an appropriate model under a desired cost-performance trade-off. Existing routing methods often estimate model suitability from the surface semantics or embedding similarity of the input query. However, such methods may ignore the underlying difficulty of a query, leading to suboptimal routing decisions. To address the challenge, we propose VDAR-Router, a difficulty-aware retrieval-based routing framework. For each input query, VDAR-Router first generates an explicit difficulty analysis. It then retrieves historical examples with similar difficulty profiles. Based on the retrieved records, it estimates candidate model suitability and selects the model using a reward function that considers both performance and cost. Experiments on three datasets show that VDAR-Router consistently achieves better cost-performance trade-offs than existing baselines. These results demonstrate the effectiveness of difficulty-aware retrieval for training-free LLM routing. Case studies further show that explicit query analysis helps retrieve more relevant examples and supports more reliable routing decisions.
Yu-Chien Tang, Jun-Chen Hung, Wen-Chih Peng et al.· 0 citations
No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.
Tao Feng, Fangxu Yu, Haozhen Zhang et al.· 0 citations
Large language models (LLMs) achieve strong performance on long-context tasks but incur substantial inference costs that scale with input length. Existing routing approaches reduce costs by dispatching queries to smaller or larger models, yet they estimate difficulty from the question text alone and overlook context-level factors such as evidence dispersion and similarity entropy. We propose the Difficulty-Aware Hybrid Router (DAHR), a lightweight framework that jointly leverages query-level, context-level, and draft-level features to classify each request into one of three inference tiers: direct answering by a small model, small-model answering with lightweight verification, or full processing by a large model. Experiments on LongBench and RULER demonstrate that DAHR reduces inference cost by 38% at the main operating point and up to 46% under stricter budgets, while maintaining competitive quality and outperforming baselines across context lengths from 4K to 128K tokens.
We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and KV-cache constraints. Unlike widely used routing heuristics that are not tied to explicit service-level objectives (SLOs) and offer limited control over latency-throughput trade-offs, we introduce a multi-objective optimization framework that formulates routing as an online linear programming with interpretable decision rewards. We apply an efficient bid-price control policy based on the online linear programming that admits requests when their SLO-weighted benefit exceeds their shadow prices. To meet millisecond decision requirements, we develop a warm-started, projected first-order updates that track the evolving dual shadow prices online with predictable runtime. We integrate our router into the Vidur simulator and demonstrate substantial improvements over standard baselines across multiple SLO regimes, including end-to-end latency, time-to-first-token, throughput, and tail performance. A big picture from our result: a science-based approach outperforms others based on heuristics.