Skip to content

Semantics, Workflows, and Infrastructure: Understanding Agent Serving at Production Scale

Sep 2026 · arXiv.org · Vol abs/2609.34432 · 1 citation · ⚡ 1 influential · 44 references
Computer Science

TL;DR

A two-week trace of 11.7 million requests from a large-scale production platform for general-purpose agents, backed by inference infrastructure comprising over 10k GPUs is analyzed to characterize the platform at three connected levels: task-level initiation semantics, workflow-level execution patterns, and infrastructure level serving demands.

Abstract

Large language model (LLM) agents execute applications through a workflow of inference requests with tool calls and user interactions. Serving these applications at production scale requires understanding how application behavior shapes inference demand and for guiding efficient execution. Recent characterization studies provide request-level workload measurements and agent execution analysis. However, an end-to-end view connecting task initiation, workflow execution, and inference infrastructure remains unexplored. In this paper, we analyze a two-week trace of 11.7 million requests from a large-scale production platform for general-purpose agents, backed by inference infrastructure comprising over 10k GPUs. We characterize the platform at three connected levels: task-level initiation semantics, workflow-level execution patterns, and infrastructure level serving demands. Our measurements reveal workload patterns such as highly skewed request volumes across sessions, rare execution overlap among logical sibling requests, and context reuse across task boundaries. Building on these observations, we analyze deployment implications and identify open problems to guide future research on agent serving systems.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Substrate-Portable Execution for Production LLM Workflows

This work validated dozens of production agent configurations across five orchestration patterns: single-inference RAG, iterative ReAct, compositional PreAct, conditional routing, and multi-agent deep research, and found no detectable difference in output quality.

T. Gopinath, Atul Kulkarni, V. Rajakumar et al. · 0 citations
Preprint Aug 2026

Bridging Agent Semantics with Spot Capacity: An Elastic and Recoverable Service Model

SemSpot is presented, a semantics-aware service model that allows agent applications to leverage the spot capacity of LLM inference platforms and develops the service model, economic boundary, and the cross-layer research agenda required to realize SemSpot.

Min-Chen Yu · 0 citations
Preprint Oct 2026

OOPMAS: Object-Oriented Multi-Agent Systems for Query-Level Workflow Generation

OOPMAS is introduced, a training-free framework that generates both the agent set and the coordination workflow at the granularity of individual queries that achieves 89.6% accuracy on a mixed-task benchmark of queries spanning code, math, and QA.

Qi Cheng, Sheng-Yu Chen, Wei Cheng et al. · 0 citations
Preprint Aug 2026

PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents

PeakBench is a benchmark of executable multi-tool workflows with execution-grounded dependency annotations and measured resource profiles that shows that strong logical planning does not reliably translate into safe or efficient execution under resource constraints, and exposes resource information to reduce avoidable...

Zhi-Kai Chen, Xu-Xiang Zhong, Song-Yan Li et al. · 2 citations
Preprint Sep 2026

Latency-Aware Orchestration for Multi-Agent LLM Workflows on Heterogeneous GPUs

A prediction-guided runtime that uses workflow forecasts to construct and optimize a physical execution graph and reduces end-to-end makespan and overall p95 completion latency under burst arrivals by up to 36.8% and 25.8% over state-of-the-art workflow schedulers.

Jing-Hao Wang, Yi-Feng Zhang, Xiao Zhou et al. · 0 citations
Preprint Oct 2026

AgentWeave: Efficient Distributed Agent Serving via Flow Decomposition

Agent applications are evolving into workflows composed of large language model (LLM) calls and tool invocations, termed modules. Existing distributed agent systems typically adopt a disaggregated architecture, deploying different modules on separate devices. While this enables pipeline parallelism, imbalanced module l...

Kai-Bin Guo, Peng-Tu Li, Zi-Cong Hong et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.