This work presents Task-to-Model Optimization (T2MO), a data-driven methodology for optimizing model selection in production coding workflows, and describes the methodology, optimization objective, evaluation protocol, and governance loop in a form suitable for production deployment and future empirical study.
Abstract
Enterprise AI coding assistants incur substantial inference spend, and naive token-cost minimization often fails to reduce end-to-end cost once retries, escalations, and developer wait time are included. We present Task-to-Model Optimization (T2MO), a data-driven methodology for optimizing model selection in production coding workflows. We treat each developer session as a task that can be discovered, classified, graded for difficulty, benchmarked in a production-like harness, and routed to the cheapest model able to complete it within quality and latency constraints. The framework is a nine-stage pipeline spanning telemetry instrumentation, taxonomy discovery, difficulty grading, benchmark construction, candidate evaluation, optimal mix derivation, forecasting and version planning, staged routing deployment, and continuous governance. Unlike token-centric routing rules, our objective is cost per completed task, with failure escalation priced in explicitly. We show that this expected-completion-cost objective weakly dominates token-cost minimization under escalation, and we derive the routing boundary, the minimum pass rate a cheaper model must reach on a given cell to be worth deploying. Decisions are organized as a two-level hierarchy of task category difficulty tier, and per-cell displacement opportunities are aggregated into a traffic-weighted savings waterfall that ranks replacement candidates by realized dollar impact. The framework supports developer guidance, spend forecasting, and a staged transition from static policies to shadow-mode classifiers, verified cascades, and ultimately an intelligent router. We describe the methodology, optimization objective, evaluation protocol, and governance loop in a form suitable for production deployment and future empirical study.
This paper introduces RSI-router, a routing framework that constructs subtask-level model assignments and model-specific skills through recursive self-improvement over accumulated experience and establishes a stronger performance--cost Pareto frontier than 9 routing methods.
Hao Li, Hang-Fan Zhang, Zhi-Yao Cui et al.· arXiv.org· 3 citations
OptiCom is proposed, a unified framework that represents LLM-driven optimizers within a shared configuration space: C=(A,Q,O,E,M,S), corresponding to artifact, query, operator, evaluation, memory, and strategy, corresponding to artifact, query, operator, evaluation, memory, and strategy.
Chen-Xing Wei, Si-Chen Liu, Lizzie Liu et al.· 0 citations
Enterprise deployments of large language models (LLMs) expose a class of operational challenges that differ fundamentally from classical machine-learning services: probabilistic outputs, prompt sensitivity, token-based costs that scale with request volume, multi-provider availability variability, and the absence of a s...
Rajesh Balaji· International Conference Inn...· 0 citations
CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance targets through per-pair independent scoring, eliminating multi-positive dilution while regularizing queries with similar routing preferences to be closer in the routing space.
Tao Yu, Yi-Fei Qu, Zhi-Qing Cui et al.· 1 citation
Multi-agent large language model (LLM) systems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost, so confidence can support initial escalation, while protocol-specific cost-aware routing remains unresolved.
Chih-Hsuan Yang, Jingyan Jiang, Cheng-Hau Yang et al.· 0 citations
Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing...
Tianyu Fu, Teng-Xuan Liu, Ruo-Xin Wang et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026