The Hidden Fragility of GNNs: How Graph Structure Amplifies Numerical Errors

Abstract

Graph Neural Networks exhibit a puzzling numerical fragility under mixed-precision training, failing significantly more often than MLPs or CNNs. This failure is inherently tied to graph structure, with heterophilic graphs and high-degree nodes being particularly vulnerable. We identify the root cause as catastrophic cancellation during neighborhood aggregation. When neighboring node embeddings point in opposite directions, their sum collapses toward zero and amplifies floating-point errors by orders of magnitude. We formalize this through the cancellation ratio ?, proving that it is fundamentally governed by graph topology, including heterophily, node degree, and network depth. Consequently, we propose Aggregation-Aware Representation Learning (AARL) to learn numerically stable and cancellation-resistant representations without sacrificing expressiveness. Unlike naive approaches that enforce neighbor alignment and destroy discriminative power, AARL maintains representation diversity while ensuring numerically safe aggregation. Experiments on diverse benchmarks demonstrate that AARL substantially improves training stability under low precision while preserving or improving classification accuracy.