This work proposes the Subject-Invariant Cross-Modal Perceived Speech Decoding (SICMD) method, which integrates functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) and conducts comprehensive analyses of the fusion method, fusion position, encoder architecture, and model inputs.
Abstract
Perceived speech decoding based on non-invasive brain-computer interface (BCI) signals has been extensively studied in recent years. Research in this field primarily faces two challenges: extracting neural representations with rich spatiotemporal information and achieving cross-subject generalization. Although separate studies have proposed methods to cope with these issues, a unified approach that simultaneously tackles both challenges remains lacking. To fill this gap, we propose the Subject-Invariant Cross-Modal Perceived Speech Decoding (SICMD) method, which integrates functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We conduct comprehensive analyses of the fusion method, fusion position, encoder architecture, and model inputs. Our results demonstrate that the proposed method improves Top-1, Top-10, and Rankacc by more than 10.6%, 10.1%, and 1.7%, respectively, compared to baseline methods in cross-subject perceived speech decoding tasks, while reducing training costs by 88.8% and 60.5% compared to multi-subject and intra-subject decoding settings. Further visualization experiments also confirm the effectiveness of our approach.
Decoding perceived speech from non-invasive brain recordings has garnered significant attention in recent years due to its wide range of potential applications. However, existing methods face considerable challenges in cross-subject decoding, primarily due to limited generalizability and the absence of explicit mechani...
Ao-Ke Zhang, Bo Wang, Xihong Wu et al.· 1 citation· ⚡1
This study presents a transparent baseline investigation of a multi-class imagined speech EEG dataset under a strictly subject-independent evaluation framework, providing a strong basis for future brain-computer interface studies targeting improved cross-subject generalization in EEG-based imagined speech decoding.
Frederik Møllskov Trier, Xiao-Peng Mao, S. Puthusserypady· arXiv.org· 0 citations
A comprehensive taxonomy of MI EEG cross-variability decoding studies from 2020 to 2025 is presented, systematically organizing advances in deep learning and transfer learning and critically evaluate core algorithmic approaches, including Convolutional Neural Networks, transformers, feature alignment, domain adaptation...
Li-Jun Wang, Yue-Ying Zhou, Peng-Pai Wang et al.· Frontiers in Neuroscience· 0 citations
Imagined speech decoding remains challenging in brain-computer interfaces (BCIs) due to the low signal-to-noise ratio and complex spatio-temporal-spectral structure of Electroencephalogram (EEG) data. Existing studies mainly rely on single-view features or simple fusion strategies, limiting their ability to capture div...
Zi-Jian Han, Zhao-Hu Liu, Honggang Liu et al.· IEEE transactions on bio-med...· 0 citations
This work introduced Dynamic Cognitive Prior Generation (DCPG), a new framework, which can be learned and is prototype based to adaptively generate priors instead of heuristic augmentations, and can be used to significantly increase the accuracy of inter-subject retrieval on the THINGS-EEG dataset with an accuracy impr...
Mehran Ali· Journal of Engineering and C...· 0 citations
This work proposes MAPA, an otherwise vanilla masked autoencoder with two spatial encodings, an anatomical region embedding and a relative positional encoding that together enable it to learn neural representations that transfer to unseen subjects and across various tasks.
Ben-Ting Tang, Z. Spalding, G. Cogan· arXiv.org· 0 citations