Honest and Reliable Evaluation and Expert Equivalence Testing of Automated Neonatal Seizure Detection

Abstract

Reliable evaluation of machine learning models for neonatal EEG seizure detection is a prerequisite for accurate performance assessment and comparison of methods. Model evaluation, however, is complicated by severe class imbalance and uncertain ground truth. To address these limitations, we propose evidence-based reporting recommendations tailored to the specific challenges of neonatal seizure detection. Using real and synthetic seizure annotations, we assessed standard performance metrics, consensus strategies, and human–expert level equivalence tests under varying class imbalance, inter-rater agreement, and number of raters. Matthews and Pearson’s correlation coefficients outperformed the area under the receiver operating characteristic curve in reflecting performance under class imbalance. Consensus types are sensitive to the level of agreement and number of raters. Among the evaluated human–expert level equivalence tests, the multi-rater statistical Turing test using Fleiss’ κ demonstrated the most consistent performance and robustness under the tested experimental conditions. Based on our findings, we recommend reporting: (1) at least one balanced metric, (2) sensitivity, specificity, PPV and NPV, (3) seizure burden agreement, (4) multi-rater agreement using the statistical Turing test with Fleiss’ κ, and (5) all of the above on a held-out validation set. This proposed framework provides an important prerequisite to clinical validation by enabling a thorough and honest appraisal of machine-learning or AI methods for neonatal seizure detection.