Skip to content

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

Sep 2026 · arXiv.org · Vol abs/2609.25624 · 1 citation · 21 references
Computer Science

TL;DR

This work presents a solution employing a set of fixed-configuration fused-upcast GEMM kernels that load 16-bit weights from memory, upcast them to FP32 in registers, and accumulate with IEEE-754 arithmetic in a reduction order that is a pure function of the problem shape and is therefore independent of the device, its SM count, or kernel scheduling.

Abstract

Large language model (LLM) outputs are expected to be reproducible under greedy decoding, yet in practice the same model, prompt, and software stack produce different outputs on different GPUs. The root cause is floating-point non-associativity combined with hardware-dependent kernel selection. Inference frameworks select different matrix-multiplication kernels on each architecture, with different parallel reduction orders and unspecified tensor-core arithmetic, and the resulting rounding differences can flip output tokens. Existing solutions have imperfect cross-architecture reproducibility and incur a significant performance penalty. We present a solution employing a set of fixed-configuration fused-upcast GEMM kernels that load 16-bit weights from memory, upcast them to FP32 in registers, and accumulate with IEEE-754 arithmetic in a reduction order that is a pure function of the problem shape and is therefore independent of the device, its SM count, or kernel scheduling. By fixing the floating-point reduction order as a function of problem shape alone, every GPU runs the same operation sequence, so cross-architecture reproducibility of the linear layers reduces to correct IEEE-754 arithmetic rather than to rounding differences staying below a tie-flip threshold. We confirm our solution's linear-layer outputs are bitwise identical across NVIDIA Ampere, Ada, and Hopper GPUs, while running $1.17$ to $3.1\times$ faster end-to-end than the state-of-the-art solution and cutting weight-memory traffic in half.

View source

Similar papers

Preprint Sep 2026

Taming Bitwise Behavior in GPU Kernels with Tensor Core: Black-Box Reconstruction, Compiler Enforcement, and Static Verification

A descriptor of GEMM reduction order is introduced, and the first black-box reconstruction of a closed-source library's arithmetic for bit-level correctness is performed, including the first black-box reconstruction of a closed-source library's arithmetic for bit-level correctness.

Zi-Teng Yang, Nicholas J. Riasanovsky, Warren Deng et al. · 2 citations
Preprint Oct 2026

Revisiting Cauchy-Schwarz Norm Bounds for SDC Detection in LLM Inference

Large Language Models (LLMs) execute billions of floating-point operations per output token across many hardware accelerators. At this scale, hardware-induced faults can bypass system-level detection mechanisms and cause Silent Data Corruption (SDC). Algorithm-Based Fault Tolerance (ABFT) is the state-of-the-art techni...

A. Altenbernd, Dylan Everingham, Philipp Wiesner et al. · 0 citations
#machine learning Preprint Aug 2026

Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance

It is found that every one of five epilogue faults -- scale precision, double rounding, multiplication order, output truncation, fused ordering -- moves the output by at most a single bfloat16 spacing, and by exactly one whenever it moves it at all, across 5,880 cells.

Teng-Ruei Chen · 1 citation
#machine learning Preprint Sep 2026

AttnFuse: A Composable DSL for Compiling Attentions to Fused GPU Kernels

Modern AI systems are built on the Transformer architecture, whose core operation, attention, accounts for the majority of computation and memory cost. Researchers continually propose new attention variants to improve quality, efficiency, or context length, but each variant currently requires expert-written GPU code to...

Varun Kumar Dasoju, Tian Zhao · 0 citations
#machine learning Preprint Oct 2026

GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference

This work presents GPU Goldmines, a framework for specifying, executing, and tuning WebGPU kernels with kernels for key LLM building blocks: matrix multiplication, matrix-vector multiplication, and flash attention, and characterize performance across diverse devices, shapes, and browsers.

Rithik Sharma, Reese Levine, Zhe-Yuan Chen et al. · 0 citations
Preprint Aug 2026

A Thread-Register Decoupled GPU Execution Model for Efficient Tensor Computation

This work proposes FIBER, a new architecture that extends the GPU SIMT (single instruction, multiple thread) model, and extends the ISA, microarchitecture, and compiler to realize shared-register addressing, conflict-free operand delivery, and fiber-based program mapping.

Zi-Han Liu, Jingwen Leng, Yang-Jie Zhou et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.