While SSL outperforms training from scratch on average and remains competitive with state-of-the-art tree ensembles, the SSL-vs-scratch gains exhibit high inter-task variance and lack significance, indicating the findings reflect general properties of tabular SSL rather than idiosyncrasies of one particular pretext task.
Abstract
Self-supervised learning (SSL) has emerged as a promising approach for tabular data, yet its efficacy under extreme label scarcity and test-time missingness remains under-explored. In this paper, we evaluate a mask-and-recover SSL pretraining objective against training from scratch and classical baselines across 14 diverse classification tasks. First, while SSL outperforms training from scratch on average and remains competitive with state-of-the-art tree ensembles (achieving ~0.8954 AUC vs. Random Forest's 0.9015 at 10% labels), the SSL-vs-scratch gains exhibit high inter-task variance and lack significance (p = 0.626 at both 5% and 10% labels). Second, contrary to the hypothesis that missing-value imputation objectives universally benefit datasets with native missingness, SSL yields the most reliable improvements on clean datasets, while frequently degrading performance on datasets with high inherent missingness. Third, despite this training variance, SSL-pretrained models achieve a higher average AUC than scratch-trained models under both test-time missingness completely at random (MCAR) injection (+0.0245 AUC, positive on 11 of 14 tasks) and structured missingness shifts (MNAR, +0.0418 AUC, positive on 8 of 14 tasks), though neither difference remains statistically significant after Holm-Bonferroni correction for multiple comparisons (adjusted p = 0.118 and p = 0.518, respectively). Fourth, comparing our mask-and-recover objective against three established tabular SSL baselines (VIME, SCARF, SubTab) under an identical encoder architecture, we find no significant difference from any of them (adjusted p = 0.459, p = 1.000, p = 1.000), indicating our findings reflect general properties of tabular SSL rather than idiosyncrasies of one particular pretext task.
C-Score, a compact framework that evaluates training behavior in three complementary spaces: prediction, feature representation, and optimization, suggests that clean accuracy alone is insufficient for evaluating SSL robustness in open-world environments, and that internal diagnostic signals are necessary for more reliable robustness assessment under unlabeled contamination.
Tsao-Lun Chen, Chicheng Fu, Han-Yi Chou et al.· 0 citations
Existing semi-supervised learning (SSL) methods rely predominantly on pseudo-labeling and consistency regularization to leverage unlabeled data, demonstrating significant performance improvements. However, we pinpoint that these methods suffer from a confidence-for-weighting issue, overvaluing high-confidence pseudo-labels while undervaluing low-confidence yet informative samples that are critical for robust generalization. In this paper, we introduce EntropyMatch, an entropy-driven SSL framework that redefines sample importance through prediction entropy rather than confidence alone. EntropyMatch employs a bidirectional weighting strategy: upward exploitation exploits reliable hard samples to refine decision boundaries while downward exploration cautiously explores uncertain ones to reduce noise. Additionally, EntropyMatch features an adaptive training mechanism that aligns with model maturity, shifting focus from safe exploration to strategic exploitation as training progresses. Experiments on eight benchmarks across various SSL tasks—spanning image classification, facial expression recognition, and human action recognition—validate EntropyMatch’s robustness and effectiveness. It consistently achieves state-of-the-art results, notably matching state-of-the-art LION’s performance on RAF-DB with just half the labeled data, demonstrating superior data efficiency and generalization.
Anyang Tong, Zenglin Shi, Zhun Zhong et al.· IEEE Transactions on Image P...· 0 citations
This article utilizes bias-variance-diversity (BVD) decomposition theory to examine the impact of noisy labels on three mainstream ensemble paradigms: Bagging, Boosting, and Stacking and characterizes the mechanisms behind performance degradation.
Guanxiong He, Jie Wang, Zhiyong Li et al.· IEEE Transactions on Neural...· 1 citation
This work conducts a comparative empirical study of five MU methods across symmetric, asymmetric, instance-dependent, and open-set noise on CIFAR-10, CIFAR-100, and the real-world noisy dataset Food-101N and finds that the appropriate unlearning strategy is conditioned on the noise structure.
Synthetic data has become a promising way to scale model training beyond limited human-generated data but it may also induce strong model collapse (Dohmatob et al., 2024), where any fixed fraction of synthetic data prevents model performance from improving under data scaling, leaving a non-vanishing excess risk floor. In this paper, we study how synthetic data affects the generalization of one-pass SGD in high-dimensional linear regression with model shift. We establish finite-sample risk bounds for mixed and two-stage training, separating standard bias and variance from source-mismatch effects, namely fluctuation and persistent drift under mixing and filtered initialization bias under two-stage. These bounds reveal a sharp contrast: mixed training induces strong model collapse, while two-stage training avoids the floor by using synthetic data only in the first stage, showing that collapse is not inevitable under a simple data curriculum. Under a random sketch model, we further obtain scaling laws for both protocols, with tight results for mixed training in the optimization-saturated regime. These laws show that larger models may amplify synthetic-induced degradation under mixing, and quantify how high-quality synthetic pretraining may reduce bias in two-stage training. Finally, we establish an exact finite-sample necessary-and-sufficient condition for two-stage training to strictly outperform real-only training under the same real-data budget and identical real-stage updates. Overall, our results highlight that synthetic data is neither inherently harmful nor beneficial; its effect depends critically on both its quality and the training protocol used to incorporate it.
This review consolidates the landscape of CP adaptations for MLL under a unified framework, examining the types of outputs and guarantees they provide, where label dependencies are incorporated, and how inference cost scales with the number of labels.
