Beyond the HBM Wall: Coherent-Memory Oversubscription for AI Workloads on NVIDIA Grace Hopper

Abstract

We realize an AI framework-agnostic interposer to test memory allocations on the NVIDIA Grace Hopper (GH200) Superchip, ranging from below the High Bandwidth Memory (HBM) limit, i.e., < 96 GB, to the HBM limit of 96 GB and beyond. When the HBM limit is exceeded, the coherent CPU+GPU memory of the GH200 can turn what would otherwise be an OOM failure on a discrete GPU into a successful but slower run on the GH200. Specifically, we characterize the performance impact of the memory footprint on the GH200 across four AI configurations: GPT-Irregular, GPT-Regular, ResNet-152, and 3D denoiser. In all cases, the highest throughput is achieved when the model fits in HBM. As the HBM limit is approached, the throughput drops off, e.g., from 8,620 tokens/second to 2,317 tokens/second (an 3.85 × decrease) for GPT-Irregular. Finally, when going beyond the HBM limit, which would result in an OOM error on a discrete GPU, GPT-Irregular successfully runs on the GH200, but the throughput drops even further to only 750 tokens/second (i.e., 11.5 × decrease).