<inline-formula><tex-math notation="LaTeX">${\sf BandPilot}$</tex-math><alternatives><mml:math><mml:mi mathvariant="sans-serif">BandPilot</mml:mi></mml:math><inline-graphic xlink:href="tang-ieq1-3716225.gif"/></alternatives></inline-formula>: Toward Performance- and Contention-Aware GPU Dispatching

Abstract

Modern multi-tenant AI clusters are increasingly communication-bound, driven by high-volume and multi-round GPU-to-GPU collective communication. Consequently, the GPU dispatcher’s choice of a physical GPU subset for each tenant largely determines the job’s effective collective bandwidth and thus its performance ceiling. Existing dispatchers predominantly rely on static, topology-aware heuristics that prioritize GPU resource compactness, assuming that minimizing physical distance maximizes communication bandwidth. However, we reveal that this assumption often fails due to complex system-level bottlenecks, such as non-linear NIC saturation and inter-node link heterogeneity. This paper presents <inline-formula><tex-math notation="LaTeX">${\sf BandPilot}$</tex-math><alternatives><mml:math><mml:mi mathvariant="sans-serif">BandPilot</mml:mi></mml:math><inline-graphic xlink:href="tang-ieq2-3716225.gif"/></alternatives></inline-formula>, a performance- and contention-aware GPU dispatching primitive that optimizes effective collective bandwidth for multi-tenant AI clusters. Specifically, <inline-formula><tex-math notation="LaTeX">${\sf BandPilot}$</tex-math><alternatives><mml:math><mml:mi mathvariant="sans-serif">BandPilot</mml:mi></mml:math><inline-graphic xlink:href="tang-ieq3-3716225.gif"/></alternatives></inline-formula> learns a data-efficient bandwidth model from sparse NCCL measurements via a hierarchical design. Guided by the model, <inline-formula><tex-math notation="LaTeX">${\sf BandPilot}$</tex-math><alternatives><mml:math><mml:mi mathvariant="sans-serif">BandPilot</mml:mi></mml:math><inline-graphic xlink:href="tang-ieq4-3716225.gif"/></alternatives></inline-formula> uses an equilibrium-driven heuristic as a fast front end, and invokes a pruned elimination search when a controller predicts that further refinement is worthwhile. To account for multi-tenant interference, <inline-formula><tex-math notation="LaTeX">${\sf BandPilot}$</tex-math><alternatives><mml:math><mml:mi mathvariant="sans-serif">BandPilot</mml:mi></mml:math><inline-graphic xlink:href="tang-ieq5-3716225.gif"/></alternatives></inline-formula> virtually merges a candidate allocation with co-located cross-host jobs to conservatively estimate shared bottleneck capacity and predict contention-degraded bandwidth. Across a 32-GPU H100 cluster and heterogeneous simulations, <inline-formula><tex-math notation="LaTeX">${\sf BandPilot}$</tex-math><alternatives><mml:math><mml:mi mathvariant="sans-serif">BandPilot</mml:mi></mml:math><inline-graphic xlink:href="tang-ieq6-3716225.gif"/></alternatives></inline-formula> achieves 90 – 97% bandwidth efficiency relative to the best-found reference, improving average efficiency by 20–30% over topology-compactness heuristics.