Contamination-Free LLM Routing on LiveBench Reasoning Tasks: Accuracy-Cost-Latency Tradeoff Learning

Abstract

Dynamic benchmarks make model selection harder and more useful at the same time: a router must identify when a low-cost policy is reliable without learning benchmark-specific artifacts. This study evaluates prompt-level routing on 200 LiveBench reasoning questions (100 zebra puzzles, 50 spatial questions, and 50 Web-of-Lies questions). Two reproducible, deterministic reasoning back ends are compared. FastLite uses exact reasoning only for recognized low-complexity cases and bounded heuristics otherwise; SolvePro uses full constraint, Boolean, and analytic spatial procedures. A router predicts whether FastLite is safe from task, length, release, constraint, and structural features available before an answer is generated. SolvePro reaches 100.0% exact accuracy, whereas FastLite reaches 67.0% exact and 69.9% answer-component accuracy. Under five-fold out-of-fold evaluation, a logistic router reaches 97.0% exact accuracy with a 32.5% strong-call rate and a 55.0% normalized cost saving relative to always using SolvePro. A transparent complexity rule preserves 100.0% exact accuracy with a 36.5% strong-call rate and 52.7% cost saving. Errors are concentrated in spatial and zebra items, confirming that prompt length alone is insufficient. The findings show that fresh, objectively scored benchmark items can support auditable accuracy-cost-latency routing when features encode verifiable computational structure. Because both back ends are deterministic policies rather than hosted LLM APIs, the reported latency and normalized cost values are controlled experimental quantities, not vendor performance claims.