Skip to content
Preprint

Every Kernel Is a Join: Automatic Multi-GPU Parallelism for AI Computations in Einsummable

Sep 2026 · 0 citations · 43 references
Computer Science

TL;DR

Einsummable is a prototype system that accepts a PyTorch-like description of an AI computation and automatically distributes it across a multi-GPU server, with no device assignments, sharding annotations, or communication operations written by the programmer.

Abstract

Distributing an AI computation across the GPUs of a multi-GPU server is one of the central problems in systems-for-AI. We present Einsummable, a prototype system that accepts a PyTorch-like description of an AI computation and automatically distributes it across a multi-GPU server, with no device assignments, sharding annotations, or communication operations written by the programmer. Einsummable models every operation as a relational join followed by an aggregation over tensor relations, in which the tuples contain sub-tensors. Each operation exposes its possible decompositions through what we call"join-agg specs". An optimizer then selects decompositions across the whole computation to minimize a communication-cost proxy. Because it searches decompositions rather than a menu of named strategies, Einsummable discovers plans that mesh-based auto-parallelizers cannot. Each decomposed operation is implemented by synthesizing an exchange program, which is a topology-aware generalization of Volcano's exchange operator. Einsummable invokes no canned collectives: all communication and aggregation is special-purpose, derived at compile time. Despite being fully automatic, Einsummable can outperform custom-designed implementations. For example, on LLaMA transformer blocks on an eight-GPU A100 server, Einsummable achieves a geometric-mean runtime of 8.97 ms, versus 13.80 ms for hand-tuned PyTorch and 15.90 ms for vLLM.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

mKernel: Fast Multi-GPU, Multi-Node Fused Kernels

Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granularity of kernels, on separate streams, reduces only part of this communication cost. Fused kernels often have better performance by transmitting each output tile as soon a...

Zi-Ming Mao, Yi-Han Zhang, S. W. Chew et al. · 1 citation
Book Open access Sep 2026

BAG: Faster Matrix Multiplication on a Single GPU

BAG (Basis Alternative Matrix Multiplication on GPUs), a GPU-oriented implementation of ABMM for the NVIDIA Ampere architecture is presented, designed to shrink workspace and eliminate redundant global-memory traffic, and introduce a cost-model-based recursion policy together with a Roofline-guided blocking strategy to...

Yao Liu, Ye-Wen Li, Zhong-Hai Zhang et al. · 0 citations
Preprint Oct 2026

PipeLink: Contract-Guided Linking of GPU Kernel Components

The fastest machine-learning kernels are compositions of components, such as loaders, matrix-multiply mainloops, epilogues and communication stages, that run concurrently on one GPU streaming multiprocessor and share its on-chip memory, barriers, warps and registers. Because these components hard-code where they sit in...

Ji-Ze Guo, Xu-Peng Miao · 0 citations
Sep 2026

Tiga: Compiling Graph Message Passing at Scale

Tiga is presented, a just-in-time compiler that separates the definition of a message-passing program from how its interactions are traversed, computed, and stored, enabling traversal specialization, fusion of relation generation with aggregation, and reverse-mode automatic differentiation.

Ming-Yuan Chi · 0 citations
#artificial intelligence Preprint Oct 2026

Cleave: Scaling Tensor Program Optimization via Decoupled Algebraic Search and Operator Scheduling

Optimized kernels such as FlashAttention and FlashDecoding are crucial for accelerating today's large models. Most of them are handwritten by experts because existing ML compilers cannot match their efficiency. Producing such kernels requires fusing computations with multiple reductions, which requires both algebraic t...

David Pissarra, Jinkun Lin, Hai-Tian Jiang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.