Skip to content
Preprint

Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast Interconnects

Nov 2026 · 0 citations · 65 references
Computer Science

TL;DR

Fast interconnects translate the per-dollar resource advantages of economical accelerators into cost-efficient long-context LLM serving, and demonstrate how low-latency, high-bandwidth interconnects enable economical accelerators to retain their cost advantage.

Abstract

Long-context large language model (LLM) serving places substantial demands on compute, memory capacity, and memory bandwidth. Economical accelerators can offer these resources at lower cost, but their limited per-device capacity requires pooling across devices, making communication overhead critical. We study cluster efficiency, measured in tokens per dollar and per joule, and show how low-latency, high-bandwidth interconnects enable economical accelerators to retain their cost advantage. Guided by a characterization of coding-agent workloads and accelerator scaling, we realize this design on the Tenstorrent Galaxy, a cluster of Blackhole chips. The serving stack combines integrated communication engines, pipelined ring transfers, communication-computation overlap, and device-side decode replay to reduce collective overhead. Across three dense models in BF16 and 8-bit configurations with 8K-256K-token contexts, the Galaxy achieves 1.07-2.24$\times$ the tokens per dollar of an eight-GPU RTX PRO 6000 Blackwell server under a three-year acquisition-and-electricity cost model. Emulating TCP-class communication on the same chips reduces throughput by 23.5-67.4% and eliminates the cost advantage in every evaluated configuration. These results demonstrate how fast interconnects translate the per-dollar resource advantages of economical accelerators into cost-efficient long-context LLM serving.

View source

Similar papers

#artificial intelligence Preprint Nov 2026

Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode

LLM inference decode executes fine-grained collectives whose latency directly limits per-user interactivity, yet scale-up fabrics are still built from electronic packet switches designed for training. We observe that, for a given model deployment and parallelization, the communication structure of decode is fixed, and...

M. Thürmer, Nicolas Scotto di Perto, Eloise Stein et al. · 0 citations

Characterizing High Bandwidth Flash for LLM Serving

Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growin...

Zack Yu, Chloe Wong, C. Hooper et al. · 1 citation
Preprint Nov 2026

Broadcast, Don't Replicate: Eliminating Weight HBM in Datacenter LLM Serving

Large language model (LLM) inference at datacenter scale is fundamentally bottlenecked by memory capacity and bandwidth limits. Because model weights are read-only and identical across instances, conventional architectures that replicate these parameters across the local DRAM of tens of thousands of accelerators incur...

Hannah Atmer, Yuan Yao, Thiemo Voigt et al. · 0 citations
Sep 2026

MaestroRAG: Orchestrated Pipeline Architecture for Efficient RAG on Edge Devices

Retrieval-Augmented Generation (RAG) offers a compact yet powerful alternative to massive Large Language Models (LLMs), making it attractive for privacy- and resource-constrained edge deployments. However, orchestrating RAG on devices with limited compute, memory, and power is nontrivial. We address this challenge with...

Deeksha Chaudhary, Cyan Subhra Mishra, Rishabh Jain et al. · 0 citations
Preprint Nov 2026

UMLoad: Fast On-Demand Model Loading on Unified Memory Architecture

On-demand model loading dominates response latency for interactive on-device AI, yet unified memory architectures (UMAs), now common on edge devices, should make it cheap by placing weights in memory the GPU already sees. However, frameworks still use the loading flow built for discrete GPUs, leaving significant ineffi...

Zi-Meng Wang, Jin-Rong Yang, Rong Chen · 0 citations
Open access Nov 2026

A Tile-Based BLAS Library for Tenstorrent Dataflow Architectures

Emerging AI accelerators increasingly adopt spatial and dataflow execution models based on distributed compute units, local memories, and explicit data movement. Although primarily designed for machine learning, these architectures are also being explored for scientific computing. This paper presents a tile-based BLAS...

Calebe P. Bianchini, E. Costa, Felipe Portella et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.