Skip to content

AgentReplay: Token-Wise Trace Replay Is Essential for Fair Serving System Performance Benchmarking

Sep 2026 · arXiv.org · Vol abs/2609.32283 · 4 citations · 54 references
Computer Science

TL;DR

The experiments show that token-wise replay in AgentReplay effectively eliminates workload variation that greedy decoding and length-wise replay cannot avoid, enabling fairer performance comparisons across serving configurations.

Abstract

LLM-based agents execute multi-turn workflows with interleaved model inference and tool calls, making efficient serving increasingly important. However, evaluating serving optimizations is challenging because identical tasks can produce different execution trajectories. Changes in generated tokens can alter subsequent prompts, tool calls, and reasoning turns, making it difficult to distinguish system improvements from workload variation. Greedy decoding does not guarantee identical outputs, while replaying only sequence lengths loses token information that affects prefix caching and mixture-of-experts (MoE) routing. To address these problems, we propose AgentReplay, a configurable trace record-and-replay framework for agent serving. AgentReplay records input/output tokens, expert selections, request dependencies, and tool durations. During replay, it forces the recorded output tokens while performing normal autoregressive computation, with optional controls for MoE expert selection and tool delays. This allows different systems to execute the same recorded workload while retaining their own batching, scheduling, and parallelization decisions. We further separate trajectory generation from performance evaluation, enabling compatible smaller models to replay long-horizon traces collected with more capable models. Our experiments show that token-wise replay in AgentReplay effectively eliminates workload variation that greedy decoding and length-wise replay cannot avoid, enabling fairer performance comparisons across serving configurations.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

TokenCast: Forecasting Token Consumption During LLM Agent Execution

TokenCast is proposed, which learns a composable cost representation for each execution segment, recording its own consumption and the context growth it introduces, and captures the extra input cost incurred when context from earlier segments is re-read by every later call.

Chao-Qian Ouyang, Ling Yue, Li-Bin Zheng et al. · 0 citations
Sep 2026

PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Serving

It is shown that pipeline parallelism (PP), long overlooked because it offers little decode-latency advantage, can reduce JCT by providing a more favorable balance between prefill and decode efficiency, and PipeSwift is built, an optimized open-source pipeline-parallel runtime integrated with a tailored micro-batch par...

Shiju Wang, Fei Ren, Fang-Cheng Fu et al. · 0 citations
#machine learning Preprint Oct 2026

TRACE: Tackling Real-World Resource Assignment Problems via Agentic Heuristic Design

Dynamic resource assignment, the real-time allocation of task streams to heterogeneous processing nodes, is the backbone of modern computing infrastructure. While learning-based schedulers excel in research, industrial deployments still rely on hand-written rules that operators can read, audit, and execute within tight...

J. Ayala-Romero, Andres Garcia-Saavedra, X. Costa-Pérez · 0 citations
Book Open access Aug 2026

Parallelizing LLM Agent Execution with Contrastive Task Allocation

Experiments on tool-augmented agent workloads show that CoAct improves per-step execution efficiency and resource utilization while achieving competitive or superior task accuracy, demonstrating that contrastive online dispatch can expose substantial parallelism in LLM-agent workflows without retraining the underlying...

Yuyang Peng, Yanling Xu, Shu-Yi Wang et al. · 1 citation

Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamics

The latency with respect to the resource dynamics of processing multiple requests and tasks concurrently is characterized and new optimization opportunities that exploit the resource dynamics of tasks are demonstrated: CPU-aware tool admission and task-aware CPU allocation.

Wonmi Choi, Minuk Park, Zhi-Xiong Niu et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.