This work proposes a two-step neural network procedure, estimating the transferable component from abundant unlabeled feature pairs that bridge the two input spaces and the non-transferable component from limited labels, and derives prediction risk bounds that improve on those of a non-transfer alternative when the non-transferable component is small or smooth.
Abstract
Pre-trained black-box predictive functions encode knowledge distilled from massive datasets and extensive computation. However, when the available input features differ from those the black box expects, direct use is infeasible. We introduce a method for transferring predictive knowledge from the black box to a new, heterogeneous input space. Our approach decomposes the target regression function into a transferable component, which the black box can inform, and a non-transferable component, which captures information unique to the new space. We propose a two-step neural network procedure, estimating the transferable component from abundant unlabeled feature pairs that bridge the two input spaces and the non-transferable component from limited labels. We derive prediction risk bounds that improve on those of a non-transfer alternative when the non-transferable component is small or smooth, and the procedure adapts to either case. Under additional conditions, the worst-case risk of our estimator is of strictly smaller polynomial order than the minimax risk of estimation from the labeled data alone. We extend the framework to multiple black boxes, each on its own input space, and show that aggregation can reduce prediction error relative to the best single black box. Simulated and real data demonstrate the practical value of the method.
Modern machine learning pipelines increasingly rely on reusing pretrained and foundation models across downstream tasks. These pretrained models can differ not only in performance but also in how they can be used: some only provide black-box predictions, while others may permit white-box access to internal representations that can be probed or fine-tuned. When deployed to the target domain in the presence of distribution shift, no single strategy, including zero-shot application, fine-tuning, or directly training a target-specific model, is uniformly the best. In this work, we propose Frontier Learning, a framework that treats a library of candidate models spanning different training histories and access regimes as complementary sources of information rather than mutually exclusive alternatives. Frontier Learning constructs a unified target-domain feature by concatenating internal representations from white-box candidates as well as prediction outputs from black-box candidates, then fits a lightweight, regularized supervised learner on this concatenated representation using labeled target data. Because the resulting hypothesis class contains predictors obtained by zero-shot reuse, fine-tuning, and direct training as special cases, empirical risk minimization over the frontier learner is guaranteed to be no worse, on the training sample, than any individual baseline. We evaluate the framework in simulations spanning varying degrees of source-target compatibility and in two real-world distribution-shift settings: visual domain adaptation on DomainNet/VisDA and clinical mortality prediction across intensive care unit domains using MIMIC-IV-Notes. Across all settings, Frontier Learning matches or outperforms the strongest individual reuse strategy, with the largest gains arising precisely when no single baseline is reliable across the range of shift considered.
Instance-wise feature selection is a valuable tool for interpreting labeled data and the predictions of black-box models. In contrast to global feature selection techniques, instance-wise methods dynamically identify important features for each instance. A growing number of methods learn a selector, which identifies important features, and a predictor, which uses these to make predictions. However, these pioneering methods face challenges including information leakage and lack of differentiability, which can slow training. In this paper, we present Hide&Seek, an end-to-end differentiable model for instance-wise feature selection. We jointly learn feature selection and prediction under a single objective without information leakage. Hide&Seek outperforms existing state-of-the-art models across a range of experiments and is fast to train. We achieve this by reformulating feature removal as a differentiable operation where instead of discretely removing features, we replace a proportion of each feature. Training is further stabilized via a parsimony-weight annealing framework.
Active Domain Adaptation (ADA) enhances transfer learning by selecting the most informative samples from the target domain for annotation. The source-free variant, ASFDA, operates using only a pre-trained source model and unlabeled target data. However, existing approaches typically assume that distribution shift arises solely from changes in the feature or label distribution, thereby neglecting concept shift (the divergence of $p(y|x)$ across domains). In this paper, we propose ADA-CS, a plug-and-play module compatible with any ADA or ASFDA framework. Without requiring access to source data, ADA-CS isolates concept shift from other forms of shifts. The concept shift severity is quantified to prioritize samples exhibiting the largest discrepancy in $p(y|x)$ for annotation. We further introduce a CSS metric to quantify the Concept Shift Severity across domains, revealing that non-negligible concept shift exists in many transfer tasks. Extensive experiments on four vision benchmarks (Office-31, Digits, DomainNet, and Office-Home RSUT) and one large-scale tabular dataset (USAccident) show that ADA-CS improves performance when combined with a wide range of active learning, ADA, and ASFDA strategies. Moreover, the rapid decline of CSS during adaptation provides direct evidence of our approach's effectiveness in identifying and correcting concept shift.
Black-box domain adaptation (BBDA) technologies have been extensively studied to transfer diagnosis knowledge from the source model to the target domain without accessing source data and source model parameters. However, existing BBDA methods assume identical fault categories across domains, which rarely holds in open and uncertain industrial environments. To solve this challenge, a novel reliable pseudolabel learning (RPLL) method is proposed for open-set black-box cross-domain fault diagnosis. In RPLL, a disentangled knowledge distillation module is designed to transfer fine-grained diagnosis knowledge from the black-box source model to the target model, effectively mitigating the impact of noisy source model predictions. In addition, an entropy-oriented label division module is designed to enhance the distinction between known and unknown fault categories. These two modules operate synergistically and achieve excellent known fault recognition and unknown fault detection via entropy-based adaptive thresholds. Extensive experiments show that the proposed RPLL achieves comprehensive diagnosis accuracies of 92.09% and 88.56% in cross-condition and cross-machine transfer tasks, respectively, significantly outperforming existing methods.
Zongzhen Ye, Qi Deng, Xiang Xia et al.· IEEE Transactions on Reliabi...· 0 citations
The experiments show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models.
K. F. Akhter, Ibna Kowsar, Manar D. Samad· Neural Networks· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
