Skip to content

Subject-Invariant Cross-Modal Decoding of Perceived Speech from Brain Recordings

Sep 2026 · arXiv.org · Vol abs/2609.30832 · 0 citations · 19 references
Computer Science Engineering

TL;DR

This work proposes the Subject-Invariant Cross-Modal Perceived Speech Decoding (SICMD) method, which integrates functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) and conducts comprehensive analyses of the fusion method, fusion position, encoder architecture, and model inputs.

Abstract

Perceived speech decoding based on non-invasive brain-computer interface (BCI) signals has been extensively studied in recent years. Research in this field primarily faces two challenges: extracting neural representations with rich spatiotemporal information and achieving cross-subject generalization. Although separate studies have proposed methods to cope with these issues, a unified approach that simultaneously tackles both challenges remains lacking. To fill this gap, we propose the Subject-Invariant Cross-Modal Perceived Speech Decoding (SICMD) method, which integrates functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We conduct comprehensive analyses of the fusion method, fusion position, encoder architecture, and model inputs. Our results demonstrate that the proposed method improves Top-1, Top-10, and Rankacc by more than 10.6%, 10.1%, and 1.7%, respectively, compared to baseline methods in cross-subject perceived speech decoding tasks, while reducing training costs by 88.8% and 60.5% compared to multi-subject and intra-subject decoding settings. Further visualization experiments also confirm the effectiveness of our approach.

View source

Similar papers

Preprint Aug 2026

Cross-Subject Generalization in Decoding Perceived Speech from Non-Invasive Brain Recordings

Decoding perceived speech from non-invasive brain recordings has garnered significant attention in recent years due to its wide range of potential applications. However, existing methods face considerable challenges in cross-subject decoding, primarily due to limited generalizability and the absence of explicit mechani...

Ao-Ke Zhang, Bo Wang, Xihong Wu et al. · 1 citation · ⚡1

Decoding Imagined Speech: A Strictly Subject-Independent Approach Using EEG

This study presents a transparent baseline investigation of a multi-class imagined speech EEG dataset under a strictly subject-independent evaluation framework, providing a strong basis for future brain-computer interface studies targeting improved cross-subject generalization in EEG-based imagined speech decoding.

Frederik Møllskov Trier, Xiao-Peng Mao, S. Puthusserypady · 0 citations
Review Open access Sep 2026

Cross-variability decoding for motor imagery EEG signals: a comprehensive review

A comprehensive taxonomy of MI EEG cross-variability decoding studies from 2020 to 2025 is presented, systematically organizing advances in deep learning and transfer learning and critically evaluate core algorithmic approaches, including Convolutional Neural Networks, transformers, feature alignment, domain adaptation...

Li-Jun Wang, Yue-Ying Zhou, Peng-Pai Wang et al. · 0 citations
Sep 2026

Contrastive Decoupling and Enhancement of Multi-view EEG Features for Imagined Speech Decoding.

Imagined speech decoding remains challenging in brain-computer interfaces (BCIs) due to the low signal-to-noise ratio and complex spatio-temporal-spectral structure of Electroencephalogram (EEG) data. Existing studies mainly rely on single-view features or simple fusion strategies, limiting their ability to capture div...

Zi-Jian Han, Zhao-Hu Liu, Honggang Liu et al. · 0 citations
Review Open access Aug 2026

Dynamic Cognitive Prior Generation for Robust EEG-to-Image Decoding

This work introduced Dynamic Cognitive Prior Generation (DCPG), a new framework, which can be learned and is prototype based to adaptively generate priors instead of heuristic augmentations, and can be used to significantly increase the accuracy of inter-subject retrieval on the THINGS-EEG dataset with an accuracy impr...

Mehran Ali · 0 citations

Pretraining for Sample-Efficient Neural Interfaces

This work proposes MAPA, an otherwise vanilla masked autoencoder with two spatial encodings, an anatomical region embedding and a relative positional encoding that together enable it to learn neural representations that transfer to unseen subjects and across various tasks.

Ben-Ting Tang, Z. Spalding, G. Cogan · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.