Fast interconnects translate the per-dollar resource advantages of economical accelerators into cost-efficient long-context LLM serving, and demonstrate how low-latency, high-bandwidth interconnects enable economical accelerators to retain their cost advantage.
Abstract
Long-context large language model (LLM) serving places substantial demands on compute, memory capacity, and memory bandwidth. Economical accelerators can offer these resources at lower cost, but their limited per-device capacity requires pooling across devices, making communication overhead critical. We study cluster efficiency, measured in tokens per dollar and per joule, and show how low-latency, high-bandwidth interconnects enable economical accelerators to retain their cost advantage. Guided by a characterization of coding-agent workloads and accelerator scaling, we realize this design on the Tenstorrent Galaxy, a cluster of Blackhole chips. The serving stack combines integrated communication engines, pipelined ring transfers, communication-computation overlap, and device-side decode replay to reduce collective overhead. Across three dense models in BF16 and 8-bit configurations with 8K-256K-token contexts, the Galaxy achieves 1.07-2.24$\times$ the tokens per dollar of an eight-GPU RTX PRO 6000 Blackwell server under a three-year acquisition-and-electricity cost model. Emulating TCP-class communication on the same chips reduces throughput by 23.5-67.4% and eliminates the cost advantage in every evaluated configuration. These results demonstrate how fast interconnects translate the per-dollar resource advantages of economical accelerators into cost-efficient long-context LLM serving.
LLM inference decode executes fine-grained collectives whose latency directly limits per-user interactivity, yet scale-up fabrics are still built from electronic packet switches designed for training. We observe that, for a given model deployment and parallelization, the communication structure of decode is fixed, and...
M. Thürmer, Nicolas Scotto di Perto, Eloise Stein et al.· 0 citations
Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growin...
Zack Yu, Chloe Wong, C. Hooper et al.· arXiv.org· 1 citation
Large language model (LLM) inference at datacenter scale is fundamentally bottlenecked by memory capacity and bandwidth limits. Because model weights are read-only and identical across instances, conventional architectures that replicate these parameters across the local DRAM of tens of thousands of accelerators incur...
Hannah Atmer, Yuan Yao, Thiemo Voigt et al.· 0 citations
Retrieval-Augmented Generation (RAG) offers a compact yet powerful alternative to massive Large Language Models (LLMs), making it attractive for privacy- and resource-constrained edge deployments. However, orchestrating RAG on devices with limited compute, memory, and power is nontrivial. We address this challenge with...
Deeksha Chaudhary, Cyan Subhra Mishra, Rishabh Jain et al.· IEEE International Symposium...· 0 citations
On-demand model loading dominates response latency for interactive on-device AI, yet unified memory architectures (UMAs), now common on edge devices, should make it cheap by placing weights in memory the GPU already sees. However, frameworks still use the loading flow built for discrete GPUs, leaving significant ineffi...
Zi-Meng Wang, Jin-Rong Yang, Rong Chen· 0 citations
Emerging AI accelerators increasingly adopt spatial and dataflow execution models based on distributed compute units, local memories, and explicit data movement. Although primarily designed for machine learning, these architectures are also being explored for scientific computing. This paper presents a tile-based BLAS...
Calebe P. Bianchini, E. Costa, Felipe Portella et al.· Anais do XXVII Simpósio em S...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.