CORAL: co-evolutionary optimization of routing and adaptive prompts for cost-efficient LLM deployment

Abstract

Deploying large language models in practice means balancing not just answer quality but also the cost and latency of producing each answer, a balance that a single fixed model rarely achieves effectively. To address this issue, two methodologies are adopted: using the best model while keeping the prompt fixed, and changing the prompt according to the query while changing the model. A model has been designed that uses the Prompt–Router Co-Evolutionary Problem (PRCP) and CORAL, which stands for Co-Evolutionary Optimization of Routing and Adaptive Prompts. It considers model selection and prompt patterns as interrelated operations and is structured to improve the performance of both simultaneously. This is achieved by maintaining two groups: one for prompts and the other for routing strategies, allowing a linked interconnection between them. This entire process is guided by multi-objective optimization techniques (NSGA-II/III) to improve correctness, cost balance, throughput, and latency. The test datasets used for evaluation include MMLU, GSM8K, and Wild Chat, which measure AI model performance. The results show that CORAL is more cost-efficient than existing methods. On GSM8K, at a matched accuracy level, it reduces cost by 31.4 percent relative to RouteLLM. At a matched cost budget, it increases accuracy by up to 4.8 percentage points, again on GSM8K against RouteLLM, and lowers the average per-query latency by 12.2 percent. Taken together, these findings suggest that treating prompt design and model selection as coupled decisions rather than independent steps offers a practical path toward lower-cost, higher-accuracy LLM deployment. The PRCP formulation introduced here provides a reusable framework for studying this coupling in future work.