Multimodal Fusion Comprehension Model for English Listening and Literature Appreciation Based on the Perceiver IO Architecture

Abstract

Traditional English multimodal listening comprehension models suffer from semantic fragmentation and weak task transferability because they lack a unified cross-modal attention mechanism, have difficulty processing high-dimensional images and long-duration speech signals simultaneously, and often rely on loosely coupled fusion strategies such as post -concatenation or simple weighting. To address these issues, this paper uses Wav2Vec 2.0 and BERT to achieve semantic alignment between speech and text, ensuring spatiotemporal consistency of linguistic information. A unified input representation structure is then constructed to improve representational consistency across modalities. A fusion encoding module centered on Perceiver IO is built, and a cross-attention mechanism is introduced to realize contextual coupling among modalities. Finally, a multi-objective prediction structure is implemented at the output layer to identify rhetorical elements, sentiment nodes, and key sentences in literary listening materials, including novels, poetry recitations, and technical literary expressions. The model achieves sentiment classification accuracy of 0.86 with F1 of 0.85, and F1 above 0.80 in cross-text rhetorical recognition. The speech-text alignment distance decreases to nearly 0.2, with BLEU ranging from 0.78 to 0.85, confirming effective multimodal fusion and comprehension.