Alzheimer’s multimodal diagnosis: enhancing cross-attention between speech and text
Abstract
Alzheimer’s disease (AD) is a complex neurocognitive disorder and the primary cause of dementia worldwide. Early and accurate detection is critical for timely intervention and improving patient outcomes. Current approaches for diagnosing dementia through spontaneous speech face significant limitations, such as the use of separately trained models, reliance on majority-vote strategies, and simple early- or late-stage fusion techniques that fail to capture deep inter-modal interactions. Furthermore, many state-of-the-art methods depend on inefficient audio-to-image transformations to utilize pretrained image models like ViT and DeiT.This study aims to develop a novel end-to-end multimodal architecture that directly integrates audio and transcript data. The goal is to eliminate the need for intermediate transformations while enhancing both intra- and cross-modal interactions for more effective AD detection. The proposed framework utilizes OpenAI’s Whisper as an acoustic encoder and BERT as a text encoder. We introduce a dual cross-modal attention mechanism to facilitate bidirectional interactions between speech and text representations. Additionally, Optimal Transport Kernel Embedding (OTKE) is employed to aggregate variable-length audio features into fixed-size representations, ensuring efficient alignment and temporal modeling. Experimental results on the ADReSS Challenge dataset demonstrate that our model achieves 90.42% accuracy, an 89.92% F1-score, and high clinical reliability with 94.53% precision and 95.00% specificity. Further evaluation on the VAS dataset confirms the model’s robustness, achieving 72.3% accuracy and a 71.2% F1-score in a three-class classification task, even when linguistic information is uniform across samples. The findings indicate that the proposed framework is a generalizable, efficient solution for early-stage dementia detection. This approach provides a robust foundation for future multimodal diagnostic research and real-world applications.