HAFT: Hierarchical Audio-Enhanced Fusion Transformer for Efficient Multimodal Sentiment Analysis

Abstract

End-to-end multimodal sentiment analysis fuses raw visual, acoustic, and textual streams whose tokenized lengths differ by up to 20:1 (1568 visual versus 77 text tokens on CMU-MOSEI). Dense attention biases fusion toward the longest modality (68% visual attention in bimodal TVLT), leaving shorter modalities—particularly the information-sparse acoustic stream—underrepresented. Current trimodal baselines consistently exhibit severe cross-category emotion imbalance (Fear F1: 25–29), a consequence of the same length-asymmetry bias that our architecture is designed to mitigate. We propose HAFT (Hierarchical Audio-Enhanced Fusion Transformer) to address three challenges arising from this asymmetry: computational bottleneck, modality imbalance, and audio under-utilization. HAFT routes audio–visual interaction through bottleneck tokens with adaptive depth-wise gating before integrating text, jointly reducing attention cost and counteracting modality bias; the Cascaded Audio Feature Enhancement (CAFE) framework strengthens prosodic representations via multi-scale time–frequency extraction; and grouped projections, decoupled positional attention, and layer-wise parameter sharing compress the remaining overhead. On CMU-MOSEI under end-to-end raw-input settings, HAFT attains [Formula: see text] Acc-2 and [Formula: see text] WF1 with 85 M parameters and 145 ms latency, improving Acc-2 and WF1 over bimodal TVLT-VA by 6.4 and 4.3 percentage points, respectively. Cross-category emotion F1 standard deviation falls from 15–17 (trimodal baselines) to 6.7, with per-category F1 range narrowing from over 43 to 16.0 points. The resulting trimodal attention distribution (33.4% visual, 32.6% audio, 34.0% text) indicates substantially reduced token-length-induced modality bias.