Cadence: Taming Coupled Contention in Disaggregated Agent Clouds
Abstract
Cadence targets multi-tenant LLM serving over operator-declared phases on disaggregated GPU pools and remote KV/cache tiers. We assume a small phase taxonomy—retrieval, tool fan-out, structured parsing, and decode—and adapt resource parameters within each phase online, rather than discovering arbitrary new behaviors at runtime. Despite strong single-tenant optimizations, clusters miss tail-latency SLOs once multi-phase requests collide with co-tenant bursts across GPU, cache, and network. The core challenge is coupled contention: a scheduler that only balances GPU utilization can still create cache hot spots and cross-rack fan-out, while transport-only fixes cannot prevent cache thrash or admission overload. Cadence jointly allocates GPU compute, KV/cache residency, and network bandwidth for phase-structured requests under per-tenant SLO and fairness constraints. It combines phase-aware resource contracts, an online co-scheduler for placement/cache/bandwidth bundles, and guardrails that combine admission control with cache and network enforcement. Across a 32-GPU, 400 Gbps leaf–spine testbed running four templates with eight tenants, at 90% offered load Cadence reduces P99 latency by 1.4–1.8 × vs GPU-Only and cuts SLO violations by 6.0–7.6 × vs GPU-Only (5.1–7.1 × vs Cache-Only/Net-Only); at the GPU-Only saturation point (100% normalized offered load), it improves cluster goodput by up to 54%. Cadence remains stable under 50% tenant churn and 40% retrieval skew, maintains Jain’s fairness index ≥ 0.84 at full load, and incurs 2.25% total CPU overhead in the control plane. The supplemental artifact includes the code, configurations, and scripts needed to reproduce the figures and tables.