This work establishes a new state-of-the-art for CI identification, with the proposed method demonstrating superior cross-dataset generalization and the power of an LLM-based multimodal framework that fuses linguistic and acoustic data to enable robust, scalable, and non-invasive screening.
Abstract
Cognitive impairment (CI) is a growing public health concern. Early and accurate diagnosis is critical for enabling timely intervention and improving patient outcomes. Speech-based CI detection has emerged as a promising non-invasive approach, as speech signals encode both linguistic and acoustic markers associated with cognitive decline. Recent advances in large language models (LLMs) further strengthen the potential of speech-based assessment by enabling more expressive representation learning and improved generalization across diverse speakers, recording devices, and clinical environments. Moreover, multimodal learning by jointly modeling linguistic and acoustic features allows for a more comprehensive characterization of cognitive and behavioral changes related to CI, leading to more reliable detection. In this work, we propose a multimodal CI detection framework based on open-source LLMs that integrates speech audio and corresponding transcripts while preserving patient privacy. Acoustic embeddings are extracted directly from speech signals, while textual embeddings are generated from automatically transcribed speech. These modality-specific embeddings are then concatenated to create a combined feature vector and used for downstream classification, without requiring access to raw or sensitive patient data. The proposed approach is evaluated on the ADReSS20 and ADReSSo21 benchmark datasets. Experimental results show that the proposed multimodal framework achieves an CI classification accuracy of 92.4% and consistently outperforms single-modality baselines. Our work establishes a new state-of-the-art for CI identification, with the proposed method demonstrating superior cross-dataset generalization. This advance highlights the power of an LLM-based multimodal framework that fuses linguistic and acoustic data to enable robust, scalable, and non-invasive screening.
Experimental results demonstrate that LSEAD provides a practical, secure, and scalable approach for early AD screening, and shows that LLM-based embeddings generalize well across datasets and improve AD classification accuracy by up to 5 percent over existing methods.
Xin Wang, Ying-lei Huang, Yuhan Su et al.· 0 citations
Early detection of cognitive impairment remains a critical public health challenge. While biomarkers such as neuroimaging and cerebrospinal fluid analyses offer high sensitivity, their limited accessibility hampers widespread screening, especially in underserved settings. Speech-based markers have emerged as promising, noninvasive indicators of cognitive decline. To develop and validate SpeechDETECT, an end-to-end speech-processing pipeline that captures fine-grained acoustic and temporal markers of cognitive impairment and provides interpretable outputs suitable for large-scale screening. SpeechDETECT comprises six modules: (1) noise reduction / amplitude normalization; (2) an eight-domain voice-analysis framework (e.g., frequency parameters, speech fluency); (3) 50 ms segment-level feature extraction; (4) feature visualization; (5) dimensionality reduction / selection (Joint Mutual Information Maximization, LassoNet, PCA); and (6) classifier training with SHapley Additive exPlanations (SHAP). Performance was benchmarked against six acoustic toolkits (e.g., GeMAPS) on two English datasets: the DementiaBank Pitt corpus (train = 166, test = 71) with single cookie-theft picture description task and NIA PREPARE Phase 2 corpus (train = 1 064, test = 267) with multiple speech tasks. A Multi-Layer Perceptron trained on PCA-derived SpeechDETECT features achieved an F1-score = 0.81% and AUC-ROC = 0.80 on the Pitt test set, outperforming the best competing toolkit (AUC = 0.76). On the PREPARE test set—comprising ≤ 30 s recordings from four speech tasks—the same model attained F1 ≈ 0.67% and AUC-ROC = 0.70, demonstrating good generalizability. Cumulative-gains analysis showed that screening the top 40% of ranked participants captured ~ 70% of cognitively-impaired (CI) cases in Pitt and ~ 63% in PREPARE. SHAP revealed speech-fluency metrics (hesitation rate, pause ratio) and high-frequency formant dynamics as the most discriminative features. SpeechDETECT delivers accurate (AUC up to 0.80) and interpretable detection of early cognitive impairment across both structured and multi-task speech settings. Its fully automated, domain-informed approach enables scalable, speech-based screening and provides a foundation for multimodal systems that combine acoustic markers with clinical or biomarker data to further improve diagnostic precision. The SpeechDETECT toolkit is openly available on GitHub at https://github.com/SpeechCARE/SpeechDETECT-Toolkit for researchers and clinicians. A demo tutorial video showing pipeline usage is available at https://github.com/SpeechCARE/SpeechDETECT-Toolkit/blob/main/SpeechDETECT.mp4.
Maryam Zolnoori, Elyas Esmaeili, Mehdi Naserian et al.· Health Information Science a...· 0 citations
The results indicate that transcript-free spectro-temporal and fluency-related cues can facilitate speaker-independent Alzheimer's disease screening from raw audio, establishing a practical foundation for deployment-oriented research.
Rashin Gholijani Farahani, A. Bastanfard· arXiv.org· 0 citations
Recent advances in Natural Language Processing (NLP) have enabled new approaches for assessing cognitive health through speech analysis. Dementia remains a growing public health concern, yet diagnosis often relies on costly and time-consuming clinical assessments. Speechbased screening offers an alternative, as language deterioration is among the earliest signs of cognitive decline. However, most existing methods fail to effectively capture both how something is said (from the audio) and what is said (from the transcribed text). Yet both components are critical for accurate and reliable diagnosis. Therefore, this work explores independent approaches to evaluate the best-performing methods. Through systematic experimentation, it was found that standard NLP preprocessing pipelines remove clinically relevant disfluency patterns from speech transcripts before modelling, discarding features that may distinguish dementia from healthy speech. To address this, the Pause-Augmented for Disfluency Markers framework was developed, a novel preprocessing approach that explicitly encodes speech disfluencies as special tokens, preserving them for consumption by the classifiers.
Speech-based Alzheimer's disease (AD) detection increasingly relies on speech-enhanced and curated versions of the Pitt Corpus, where speech enhancement, sample selection, and demographic balancing are often treated as beneficial preprocessing steps. However, whether these transformations improve real-world AD detection or instead affect model generalization and prediction behavior remains unclear. In this work, we revisit the role of speech preprocessing and dataset curation across widely used benchmarks for speech-based AD detection. We evaluate the speech quality of different datasets, the cross-dataset generalization of multiple deep learning models under matched and mismatched enhancement settings, and the behavior of several recent large audio-language models (LALMs). Experimental results show that across multiple supervised speech models, speech-enhanced datasets often improve in-domain performance while reducing robustness in cross-domain evaluation. Matched enhancement between training and test data alleviates, but does not eliminate, this degradation. LALMs show a similar sensitivity: enhanced datasets induce stronger class imbalance and prediction shifts than unprocessed data. These results suggest that speech preprocessing and dataset curation can substantially influence downstream AD detection behavior, indicating that ``cleaner''speech datasets are not necessarily more reliable for real-world AD detection.
Depression is among the leading causes of disability worldwide, yet its detection continues to rely heavily on subjective clinical interviews and self-report instruments that are difficult to scale, particularly in low-resource regions. This paper presents the design, implementation, and empirical validation of a multimodal fusion neural network for automatic depression detection from speech and text, developed with specific attention to closing the near-total absence of African research contributions in this rapidly growing area of computing. The proposed system extracts spectral-prosodic acoustic features from voice recordings using a convolutional encoder, and contextual semantic features from transcribed text using a transformer-based language encoder, before an attention gate learns to weight the relative contribution of each modality per instance ahead of a fully connected classifier. The architecture was implemented and validated end-to-end as a feasibility study on two accessible, weak-label proxy corpora: the RAVDESS acted-emotion speech dataset, with sad and calm recordings relabeled as a depression-like class, and the dair-ai/emotion text corpus, with sadness and fear posts relabeled likewise. On held-out test data, the fusion model reached 90.28% accuracy and 0.9609 AUC-ROC, and, most notably, recovered the positive-class recall that the audio-only model lost almost entirely (21.05% versus 84.21%), demonstrating that the attention-gated fusion mechanism functions as designed. This paper reports the background, problem definition, related work, the complete mathematical formulation of the implemented pipeline, the empirical results of this proxy validation, and a discussion of what they do and do not establish, closing with concrete recommendations for advancing the work toward a clinically meaningful, Africa-relevant screening tool.
Bulus Gago Kindness, Yebduya Garleya Lodiya, Kamak Yamlach Shedrach et al.· International journal of res...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
