Jul 2026· IEEE Transactions on Pattern Analysis and Machine Intelligence· Vol PP, pp. 1-16· 0 citations
Medicine
TL;DR
Label Wave is proposed, which does not require validation data for selecting the desired model across various weakly supervised learning paradigms, including learning with noisy labels (LNL), positive-unlabeled learning, and unlabeled-unlabeled learning.
Abstract
Early stopping is a fundamental technique for mitigating overfitting in machine learning and is especially critical in many weakly supervised learning scenarios. Conventional early stopping, however, needs a sufficiently sized and representative validation set. This creates a trade-off: dedicating more data for validation can diminish the training set and limit model performance, while an insufficient validation set may lead to sub-optimal model selection. This challenge is particularly serious in many weakly supervised learning scenarios where validation data may be entirely unavailable or unreliable. In this paper, we propose Label Wave, which does not require validation data for selecting the desired model across various weakly supervised learning paradigms, including learning with noisy labels (LNL), positive-unlabeled learning, and unlabeled-unlabeled learning. It works by tracking the changes in the model's predictions on the training set during the training process, aiming to halt training before the model excessively fits misleading patterns. This method is empirically supported by our observation that minimum fluctuations in predictions typically occur at the training epoch before the model transitions from fitting reliable patterns to misleading ones. Extensive experiments further provide large-scale evaluation for LNL across diverse noise conditions, model families, optimizers, and data modalities. Label Wave selects near-oracle checkpoints (test-accuracy oracle gaps: 0.15-0.93 percentage points in controlled sweeps and 1.90 percentage points on 71 tabular tasks) and boosts 7 LNL methods by 1.98/2.86 percentage points over 10%/20% hold-out validation.
C-Score, a compact framework that evaluates training behavior in three complementary spaces: prediction, feature representation, and optimization, suggests that clean accuracy alone is insufficient for evaluating SSL robustness in open-world environments, and that internal diagnostic signals are necessary for more reliable robustness assessment under unlabeled contamination.
Tsao-Lun Chen, Chicheng Fu, Han-Yi Chou et al.· 0 citations
Estimating the generalization error (GE) of machine learning models is fundamental, with resampling methods being the most common approach. However, in nonstandard settings, particularly those where observations are not independently and identically distributed, resampling using simple random data divisions may lead to biased GE estimates. This paper strives to present well-grounded guidelines for GE estimation in various such nonstandard settings: clustered data, spatial data, unequal sampling probabilities, concept drift and hierarchically structured outcomes. Our overview combines well-established methodologies with other existing methods that, to our knowledge, have not been frequently considered in these particular settings. A unifying principle among these techniques is that the test data used in each iteration of the resampling procedure should reflect the new observations to which the model will be applied, while the training data should be representative of the entire data set used to obtain the final model. Beyond providing an overview, we address literature gaps by conducting simulation studies and a real data study. These studies assess the necessity of using GE-estimation methods tailored to the respective setting. Our findings corroborate the concern that standard resampling methods often yield biased GE estimates in nonstandard settings, underscoring the importance of tailored GE estimation.
R. Hornung, Malte Nalenz, Lennart Schneider et al.· Statistical Science· 0 citations
It is shown that a usable signal is available after convergence, when loss no longer distinguishes the two populations, and applying a fixed perturbation to a converged model's inputs flips the predictions of the latter far more often than the former.
This work shows that early stopping can overcome suboptimality: in a Gaussian mixture model with label-flipping noise, GD stopped at an appropriate oracle time achieves minimax-optimal excess zero-one risk for covariance spectra with fast and continuous decay, including polynomial and exponential spectral decays.
Alex Buna, S. Liu, Patrick Rebeschini· 0 citations
The problem of overfitting is one of the most persistent in the current machine learning (ML), especially as models and data dimensionality increases. Although classical regularization methods like L1, L2, dropout, and early stopping have been proven to be efficient, their weakness is realized in large-scale, deep, and data-sparse learning settings. In the current paper, a detailed study has been carried out on new regularization techniques that aim to enhance the performance of generalization and, at the same time, ensure the expressiveness of the model. We present a single taxonomy of the new regularization methods such as adaptive regularization, information-theoretic constraints, structured sparsity, stochastic regularization and regularization at the representation level. Moreover, we suggest Hybrid Adaptive Information Regularization (HAIR) which is a dynamic complex/generalization balance which is regularized by entropy-based penalties and parameter-sensitivity analysis. Numerous comparative studies show that the suggested approach is more effective than the traditional approaches in various learning paradigms. The findings have emphasized the importance of advanced regularization in developing robust, scalable and interpretable ML systems. The current study provides a certain contribution to both theoretical background and methodological developments as well as empirical findings in favor of next-generation regularization approaches.
Rak esh, A. An· International Journal of Mac...· 0 citations
Selecting a single best machine-learning regression model from a set of competing models can be challenging. While models selected based on cross-validation performance do not guarantee good predictions on external data, models selected solely on external validation performance do not ascertain precise predictions for other external sets. Therefore, we propose three quantitative metrics to guide modelers in selecting the best model using the modeling set information only. Three quantitative data sets of varying sizes and complexities were considered. Each data set was randomly split into a modeling set and an independent test set. The modeling set was further split thrice to generate training and validation sets. Various machine-learning models were developed and validated against the validation sets. Our proposed metrics were computed for each model using only training and validation set performances. The novel metric values guided the selection of the best models, which also demonstrated expected performance on the independent test set. Furthermore, successful applications of our framework on two additional benchmark data sets demonstrated its wider generalizability.
Arkaprava Banerjee, K. Roy· Journal of Chemical Informat...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
