PREreview of "Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing"

Abstract

This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/23147109. ## Summary Routing evaluations start the clock too late: they measure serving-time savings but ignore the upfront supervision cost — up to N×M query–model executions before deployment. The paper introduces supervision-amortized metrics: SA-BEP (deployment queries needed for serving savings to recover supervision spend, Eq. 1) and SA-CR (cost ratio after amortizing over horizon H). SaveRouter learns from sparse feedback via query grouping, adaptive capability–uncertainty acquisition (UCB, Eq. 2), a hierarchical group–model capability estimator (two-way additive prior + shrinkage), and a query-level residual correction. On four benchmarks (LLMRouterBench, Mixinstruct, MMR-Bench, RouterBench) it uses only 33–41% of training feedback while matching or beating fully supervised routing quality, cutting break-even volume ~1.9–9.5× vs. the fastest conventional router (e.g., LLMRouterBench: 5.3K vs. 11.5K queries; RouterBench: 42.6K vs. 326.4K). ## Strengths 1. **The economic framing is the contribution, and it's correct.** Eq. 1 makes explicit what practitioners feel but papers ignore: identical serving efficiency can mean wildly different payback if one router cost 10× more to build. The "decision sufficiency vs. matrix recovery" distinction (§3.2) — routing needs argmax[y − λc], not the full quality matrix — is crisp and should change how people budget supervision. Figure 1(b) (EmbedLLM recovers 99% of full-supervision accuracy at 30% supervision) is a genuinely useful empirical fact. 2. **Honest reporting of inconvenient results.** Table 2 shows Random-K reaching break-even *earlier* than SaveRouter (3.8K vs. 5.3K) — reported and correctly interpreted (earliest payback ≠ long-horizon efficiency). Papers that publish the result complicating their story earn trust. 3. **Serious evaluation breadth.** Four heterogeneous benchmarks, ten baselines through one ORBIT pipeline with identical splits, ablations of both estimator and acquisition, a supervision-budget sweep (Figure 3), and paper-derived reimplementations disclosed as such. ## Major concerns 1. **C0 excludes the most expensive cost: labels.** Supervision cost charges only model executions — "router training itself is not included in C0" (Appendix A.1), and per-pair costs come pre-computed from the benchmark. In practice, quality labels for open-ended tasks need LLM judges or humans, and the judging bill often dominates execution. The payback horizons describe a world where labels are free once the model runs; a judge-pricing sensitivity analysis would test whether the 1.9–9.5× advantage survives a real labeling pipeline. 2. **On some benchmarks, the honest conclusion is "don't route."** Mixinstruct needs 1.23M deployment queries for even SaveRouter to break even; on MMR-Bench most methods sit at CR = 1.0000 and SA-BEP = ∞ — never beating the best single model. The paper's own metrics produce this conclusion but never state it. 3. **Acquisition is simulated on a static matrix.** "Adaptive acquisition" reveals entries of a pre-computed matrix under a fixed 20/80 in-domain split (seed 42) — no distribution shift, no model-pool churn, no price changes. But the economic argument is about real deployment, where the common event is a new model joining the pool or a price change invalidating the trade-off. Pool-churn is untested, and the paper has no limitations section at all — a notable absence for deployment-economic claims. ## Practitioner perspective "Will this router ever pay back what it cost to build?" is the question I ask before any routing project, and SA-BEP is a metric I will actually use. But the production C0 includes the judge pipeline, the eval harness, and engineer time — none on this paper's books. My rule of thumb: if projected deployment volume is under ~10× the SA-BEP computed on your own full cost basis, skip the router and ship the best single model. ## Overall assessment A valuable paper whose economic framing (SA-BEP/SA-CR, decision sufficiency) genuinely advances how routing should be evaluated, with unusually honest reporting. Recommend with revisions: include realistic labeling costs in C0 (or a sensitivity analysis), state the "don't route" implication of the Mixinstruct/MMR-Bench numbers directly, test pool-churn rather than only static-matrix acquisition, and add the missing limitations section. Competing interests The author declares that they have no competing interests. Use of Artificial Intelligence (AI) The author declares that they used generative AI to come up with new ideas for their review.