ProgRouter is presented, an online progress-guided routing framework that adaptively selects LLM agents across workflow steps to preserve task-solving quality while adhering to time and cost budgets and reduces the operating cost relative to key baselines while maintaining strong task-solving performance.
Abstract
Multi-agent large language model (LLM) workflows have emerged as a powerful paradigm for solving complex, open-ended tasks through collaborative reasoning among specialized LLM agents, but they incur substantial operating costs due to repeated LLM invocations and long-horizon context accumulation. Existing cascade routing methods make one-shot, query-level decisions and cannot adapt to the dynamic, state-dependent nature of multi-step workflows, in which the right LLM at each step depends on evolving task progress, remaining task difficulty, and cost-efficiency requirements. We present ProgRouter, an online progress-guided routing framework that adaptively selects LLM agents across workflow steps to preserve task-solving quality while adhering to time and cost budgets. ProgRouter introduces a multi-view task progress scorer that combines coarse workflow outcome regimes with fine-grained signals on subtask completion, progress trends, and workflow state quality. Then, a dual-path task progress predictor and an adaptive meta-gating mechanism estimate the progress gain for each candidate routed LLM. ProgRouter makes online step-wise routing decisions that balance progress gain, task time budgets, and long-term operating cost efficiency. Experiments on HumanEval Plus, MBPP, MATH-500, and ASQA, spanning agentic code generation, mathematical reasoning, and retrieval-augmented long-form question answering, demonstrate that ProgRouter reduces the operating cost relative to key baselines while maintaining strong task-solving performance.
EASy is proposed, a trainable agentic framework that jointly optimizes task performance and computational efficiency through reinforcement learning and consistently achieves stronger performance-efficiency trade-offs than strong agentic baselines.
Junnan Liu, Linhao Luo, Thuy-Trang Vu et al.· 0 citations
A Task-Oriented Prefix-Aware Scheduler that jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution and scores candidate post-decision states by trading off the expected reduction in each task's longest remaining service path against the near-term benefit of downstream prefix reuse.
Hongqiu Ni, Han Tian, Chi Zhang et al.· 1 citation
Complex tasks often decompose into parallelizable yet interdependent subtasks, making orchestration critical to the performance of multi-agent systems (MAS). Existing evaluations typically rely on end-to-end execution, which conflates orchestration-plan quality with worker capabilities, tool reliability, and environmental noise. Moreover, the time and token costs of real execution grow rapidly with workflow scale, making systematic evaluation expensive. We present OrchBench, a simulation-based benchmark for evaluating multi-agent orchestration plans in isolation. Starting from real-world tasks, OrchBench constructs directed acyclic graphs (DAGs) that encode task dependencies, with controlled sizes and degrees of parallelism. Given a DAG, a per-agent context limit, and an agent budget, the evaluated planner assigns subtasks to agents and specifies cross-agent information transfers and their retention ratios. A deterministic simulator evaluates the resulting plan without invoking worker agents and returns interpretable measures of result quality, makespan, and token cost. The simulated scores produced by OrchBench correlate strongly with quality scores from Claude Code executions, achieving a Pearson correlation of \(r=0.816\), while requiring only \(1.3\%\) of the tokens and \(10.3\%\) of the wall-clock time. Across diverse planners and workflow scales, we find that preserving task-critical information is more important than simply increasing the number of agents, and the benefits of parallelism diminish as coordination failures accumulate. These results establish OrchBench as an efficient and interpretable benchmark for comparing and diagnosing multi-agent orchestration plans.
LLM agents solve complex tasks by executing multi-step workflows that interleave LLM inference with external tool calls, yet execution efficiency is often the dominant bottleneck in real deployments because LLM-generated workflows are typically chain-structured and inherently sequential, limiting parallelism and underutilizing available compute resources. We propose CoAct, a training-free framework that parallelizes agent workflows by casting execution as an online task allocation problem: CoAct prompts the LLM to generate a pool of discrete subtasks and performs online dispatch by selecting, whenever a worker becomes available, the next task that minimizes an incremental task-contrastive objective, encouraging high similarity among tasks executed on the same path (positive pairs) and low similarity across different paths (negative pairs) to reduce cross-worker interaction and synchronization. CoAct further supports speculative redundancy via selective re-execution to improve robustness and mitigate tail latency. Experiments on tool-augmented agent workloads show that CoAct improves per-step execution efficiency and resource utilization while achieving competitive or superior task accuracy, demonstrating that contrastive online dispatch can expose substantial parallelism in LLM-agent workflows without retraining the underlying model.
Yuyang Peng, Yanling Xu, Shu-Yi Wang et al.· Proceedings of the 32nd ACM...· 1 citation
Large Language Model-based Multi-Agent Systems (LLM-MAS) have shown exceptional promise for complex tasks, including retrieval-augmented generation and autonomous data analytics. However, their deployment in resource-constrained industrial environments faces critical challenges, such as unpredictable end-to-end latency, low task success rates, and expensive operational costs. Existing orchestration strategies for LLM-MAS mainly rely on static heuristics, implicit LLM-based routing, or reinforcement learning, which suffer from brittleness under workload drift, prohibitive online exploration costs, and inherent governance risks. To address these issues, we propose G-STAR, a general graph-based scheduling framework that formalizes complex MAS pipelines as attributed Directed Acyclic Graphs (DAGs). Specifically, G-STAR decouples its learning pipeline into three critical stages to ensure both system stability and execution efficiency. First, we build a data layer that logs fine-grained signals for agent node (e.g., execution latency, token I/O, model IDs, resource metrics) and task-grounded outcomes (e.g., accuracy, cost, SLA), yielding a trace-driven dataset. Second, we train an offline GNN model to predict optimal dynamic routing decisions, including agent activation, concurrency-constrained Top-K selection, and execution ordering, conditioned on the evolving graph state. Finally, a lightweight dispatcher deploys the pre-trained GNN model for single-forward-pass routing with negligible computational overhead. Extensive experiments on two public benchmarks and our real-world production workload confirm G-STAR's superiority over static and LLM-driven baselines. It maintains competitive task quality while reducing latency variability and consistently improving end-to-end latency. Furthermore, we develop an industry-grade orchestration stack with asynchronous execution, resilient serving, and audit-friendly artifacts, offering a practical solution for optimizing web-scale deployments of complex MAS pipelines.
Jiabao Song, Yunsheng Xia, Beibei Kong et al.· Proceedings of the 32nd ACM...· 0 citations
Agentic LLM workflows consist of sequences of model turns interleaved with tool interactions, so their end-to-end completion time depends not only on inference speed but also on when ready turns are released. Most runtimes release each turn immediately upon readiness. Under contention, this eager release policy can accumulate released but unfinished work; once submitted, those turns can no longer be reordered by the workflow-level policy, increasing tail latency. We present a tail-risk-aware turn release scheduling method that jointly decides which ready turn to release next and how much released but unfinished work to maintain. The method uses a mean--Conditional Value-at-Risk (CVaR) objective to capture the evolving tail risk of unfinished workflows, incorporates online estimates of turn work when prioritizing ready turns, and adapts the released work budget to observed queue pressure. We evaluate the method using real agent execution traces from software engineering tasks across multiple LLMs and workflow arrival rates. The method performs comparably to eager release under light load and substantially reduces the P95 of workflow flow time under contention, achieving up to a \(3.50\times\) speedup.
Bochao Feng, Jianjiang Li, Haojie Wang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
