Orchestrating Gradients Hierarchically via Utility Equilibration for Robust Multimodal Learning

Abstract

Modality imbalance remains a central challenge in multimodal learning. The dominant modality suppresses weaker encoders, and the resulting model degrades sharply once a modality becomes noisy or absent. We trace the pathology to the fusion module, where two gradient defects arise, Magnitude suppression and Direction entanglement. Since both defects vary with network depth, per-modality rebalancing methods that apply a uniform correction across layers only partially address them. We propose Orchestrating Gradients Hierarchically (OGH), which intervenes at every encoder layer through a single adaptive scalar~?. OGH-M applies a geometric magnitude schedule that gives shallow layers the largest gradient amplification; OGH-D attaches lightweight bypass classifiers and replaces the contaminated gradient with a clean, encoder-only signal. An adaptive calibration procedure profiles layer-wise gradient utility periodically and sets~?. On seven benchmarks spanning audio-visual, sentiment, and action recognition tasks, OGH ranks first on four and is competitive on the rest, with gains up to +6.98\,pp on KineticsSounds and +5.24\,pp on CREMA-D over ten recent baselines. The per-layer balancing also improves robustness under modality degradation (e.g.\ +8.95\,pp under audio missing) without any robustness-specific training objective.