Measuring Evaluation Leakage in EEG Motor Imagery Decoding, with an Application to Stroke Patients

Abstract

Brain-computer interfaces (BCIs) that decode imagined movement from EEG are often evaluated using Common Spatial Patterns (CSP), a supervised filtering method. If CSP is fitted to all data before splitting into training and test sets, information from the test labels leaks into the model and inflates reported accuracy. This study measures the size of that effect. In a null experiment where true accuracy should be at chance (50%), fitting CSP before the split produced 91% within-subject accuracy, a 41-percentage-point inflation, compared with about 5 points in cross-subject evaluation. On real data from 87 healthy participants and 46 post-stroke patients, leakage added 14.3 and 10.6 points respectively. Adaptive signal amplification was also tested as a way to improve decoding for stroke patients and showed no significant effect, a result confirmed with a Riemannian classifier. These findings show that correct evaluation protocol matters as much as model choice when reporting BCI performance.