COVER-EL: Class cOVERage-Aware Noise Correction for Crowdsourced Labels via Elimination-Based Inference.

Abstract

In crowdsourcing scenarios, where each instance is labeled with multiple noisy labels, its true label is estimated by combining label integration and various recently proposed noise correction methods. Recent correction methods typically partition the data into clean and noisy sets, then train classifiers on a high-consistency clean set to relabel the remaining noisy instances. However, under high residual noise after label integration, these methods often lead to insufficient class coverage, where some classes are severely underrepresented or even absent in the clean set, making subsequent correction unreliable and bias-prone. To address this issue, we propose a class coverage-aware noise correction method for crowdsourced labels via elimination-based inference (COVER-EL). At first, COVER-EL estimates instancewise label confidence from multiple noisy labels to construct an initial clean set. Then, one-versus-rest (OvR) binary subpredictors are trained with adaptive confidence thresholds to ensure reliable predictions. For each label-ambiguous instance, COVER-EL forms a candidate label set from the distinct crowd-provided labels and iteratively eliminates ambiguous candidates using reliable classwise subpredictors, with predictor-label refinement repeated until stabilization. Theoretically, we provide a probably approximately correct (PAC)-style bound that gives a high-probability guarantee on the overall error after correction. Experimental results on 34 simulated datasets and two real-world crowdsourced datasets show that COVER-EL outperforms existing state-of-the-art correction methods.