Theoretical Analysis of Measure Consistency Regularization for Partially Observed Data

Abstract

The problem of corrupted data, missing features, or missing modalities continues to plague the modern machine learning landscape. To address this issue, a class of regularization methods that enforce consistency between imputed and fully observed data has emerged as a promising approach for improving model generalization, particularly in partially observed settings. We refer to this class of methods as Measure Consistency Regularization (MCR). Despite its empirical success in various applications, such as image inpainting, data imputation and semi-supervised learning, a fundamental understanding of the theoretical underpinnings of MCR remains limited. This paper bridges this gap by offering theoretical insights into when MCR yields a more favorable finite-sample estimation-error upper bound, viewed through the lens of neural network distance. Under ideal interpolation and compatibility conditions, we show that the MCR estimation-error upper bound is no larger than vanilla supervised training and becomes strictly smaller when the mixed-sample alternative is favorable. We then extend the analysis to the non-ideal regime, where optimization and compatibility residuals can potentially offset this finite-sample advantage. Guided by these insights, we propose a novel practical diagnostic that leverages the duality gap and a calibrated drift estimator to infer the potential benefit of MCR training. We present detailed empirical evidence to support our theoretical claims and to show the effectiveness and accuracy of our practical diagnostic. We further provide simulations on real-world datasets to show the versatility of MCR under different model architectures designed for different data sources.