A two-week trace of 11.7 million requests from a large-scale production platform for general-purpose agents, backed by inference infrastructure comprising over 10k GPUs is analyzed to characterize the platform at three connected levels: task-level initiation semantics, workflow-level execution patterns, and infrastruct...
Yi-Hao Zheng, Jing-Zhe Jiang, De-Jiang Zhu et al.· 0 citations
PackServe is a scheduler designed to reduce resource costs while meeting latency SLOs for agentic LLM serving, which uses compact white-box models to predict latency under prefill/decode interference and packs requests onto fewer serving instances while preserving KVC reuse and SLO constraints.
Zhi-Yuan Tan, De-Jiang Zhu, Jing-Zhe Jiang et al.· 0 citations
Online image generation with Diffusion Transformers (DiTs) must meet latency service-level objectives (SLOs) while using GPU resources efficiently. Existing systems improve GPU utilization by batching multiple requests for joint execution. However, request-level batching offers limited control over batch size: batches...
Zhexiang Zhang, Min-Chen Yu, Yi-Fan Sun et al.· 0 citations
SemSpot is presented, a semantics-aware service model that allows agent applications to leverage the spot capacity of LLM inference platforms and develops the service model, economic boundary, and the cross-layer research agenda required to realize SemSpot.
Min-Chen Yu· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
