Natural language decoding from EEG via transfer learning and multimodal contrastive learning
TL;DR
A framework for sentence-level language decoding based on transfer learning and multimodal contrastive alignment is proposed, training an EEG encoder to align neural signals with pretrained text embeddings that are subsequently inverted into text using a pretrained embedding-to-text model.
Abstract
Decoding natural language from non-invasive electroencephalography (EEG) is a key step toward practical brain–computer interfaces for individuals with speech impairments. While prior work on the Large Spanish Speech EEG Dataset has focused on sentence classification, semantic reconstruction remains largely unexplored. We propose a framework for sentence-level language decoding based on transfer learning and multimodal contrastive alignment, training an EEG encoder to align neural signals with pretrained text embeddings that are subsequently inverted into text using a pretrained embedding-to-text model. Under 10-fold subject-wise cross-validation, our approach achieves a BERTScore of 0.320 ± 0.021, BLEU-4 of 0.249 ± 0.026, and 21.7 ± 0.031% sentence classification accuracy over 30 classes. Ablation results indicate that reconstruction performance is primarily driven by contrastive alignment. These findings demonstrate the feasibility of semantically meaningful sentence reconstruction from non-invasive EEG.