Distributional mismatch between the data used to construct a statistical procedure and the population to which it is ultimately applied is pervasive in modern data analysis. We study covariate shift, a fundamental instance of this problem, and develop an adaptive importance-weighted model averaging method for prediction when labeled observations are available from a source distribution, whereas only unlabeled covariates are observed from the target distribution. Procedures fitted directly to the source sample generally optimize prediction risk under the source distribution and may therefore be suboptimal for target prediction. Importance weighting by the density ratio between the target and source covariate marginals provides a natural correction, but a small number of large density-ratio values can substantially inflate the variance of the resulting estimator in finite samples. We address this bias-variance trade-off by treating the degree of importance-weighting correction as a source of model uncertainty. Specifically, we construct a family of adaptive importance-weighted least-squares estimators by raising the estimated density ratio to a range of exponents, with the endpoints corresponding to ordinary least squares and standard importance-weighted least squares, and form a data-driven average over these candidates. Under model misspecification, the proposed model averaging estimator is shown to be asymptotically optimal relative to the infeasible best convex combination of the candidate estimators. Under correct specification, a diverging penalty is shown to make the selected weights concentrate near the ordinary least-squares endpoint. Simulations and a real-data application show that the proposed method achieves competitive target-prediction performance across the settings considered.
This paper develops procedures for nonparametric goodness-of-fit testing under covariate shift, where labelled data are drawn from a source population but goodness-of-fit is evaluated for a target population. The distribution mismatch is quantified by either a bounded moment condition or a sub-exponential tail condition on the target-to-source density ratio. Our method combines truncated importance-weighting kernel ridge regression with a multiplier bootstrap to construct confidence sets for the regression function. The truncation stabilizes the importance- weighting kernel ridge regression as well as the bootstrap calibration, making our approach applicable even when the density ratio has heavy tails. We prove nonasymptotic validity and sharpness of the resulting confidence sets under suitable operator compatibility conditions, and establish explicit error rates for coverage probability under specific conditions on the target- to-source density ratio and on the spectral decay of the kernel integral operator. Numerical experiments corroborate our theoretical findings.
Reweighting source samples to match a target covariate distribution is a standard response to distribution shift when generalizing evidence from one population to another. This strategy is well suited to deterministic, learnable covariate discrepancies, but can be insufficient when source--target population differences also contain changes beyond covariate shift or when estimation of the density-ratio weights is unstable. To address this challenge, we introduce a new model that allows non-systematic changes between two population laws after systematic shifts are accounted for. Such residual shift is modeled as random perturbations to the probability space that cannot be represented in a learnable way. In this way, we separate systematic shifts, treated as bias and corrected by reweighting, from residual random perturbations, treated as distributional uncertainty and handled through dataset pooling. Under pure random perturbations, this principle yields Augmented Inverse Distance Weighting (AIDW), which uses regression augmentation and variance-optimal dataset-level pooling. For mixed shifts, we develop Augmented Inverse Hybrid Weighting (AIHW), which interpolates between AIDW and standard augmented importance weighting. Both methods trade off sampling uncertainty and distributional uncertainty via a \emph{distributional distance} that describes the strength of random perturbations. We establish asymptotic properties of the methods, together with plug-in guidance for choosing tuning parameters and model diagnostic tools. Experiments on three real-world multi-site datasets demonstrate consistent reductions in mean-squared error compared with standard weighting baselines, along with substantially improved empirical coverage in settings where covariate-shift adjustment alone undercovers, showing the robustness of the proposed methods across diverse distribution shift scenarios.
Average dose-response functions are widely used to summarize causal effects of continuous treatments, but most existing methods assume that the observed sample represents the target population. We study a covariate-shift setting in which covariates, treatment, and outcome are observed in a labelled source sample, while only covariates are observed in the target sample. We develop a two-sample local polynomial regression framework based on pseudo-outcomes that use source outcomes to address confounding and target covariates to define the population of interest. We further propose a source-to-target extension of distance covariance optimal weighting (DCOW), designed to remove treatment-covariate dependence in the source sample while aligning the weighted source covariate distribution with the target population. A central theoretical contribution is a weight-level analysis of this optimization-based procedure: we show that the population criterion identifies the oracle source-to-target weights and that approximate empirical minimizers, including exact minimizers as a special case, converge uniformly to these weights under regularity conditions. We also establish consistency and asymptotic normality of the resulting estimator. Simulations show that the proposed method improves target dose-response estimation relative to DCOW, generalized-propensity-score weighting, entropy balancing, and unweighted alternatives. We illustrate the method in a county-level analysis of PM2.5 exposure and subsequent heart-disease mortality using a source-target validation design.
Covariate shift often occurs because, in many real applications, the source and the target observations may be generated from different distributions. In this case, the standard metric under the source distribution is not appropriate. This paper considers deep neural network estimators for nonparametric quantile and Huber regression under covariate shift and from dependent observations. We deal with a generalized Bernstein-type inequality that is satisfied by many classical models, including i.i.d. observations, $\phi$-mixing, strong mixing, and $\mathcal{C}$-mixing processes. To perform the covariate shift phenomenon, we propose a sparse-penalized deep neural network (SPDNN) estimator that takes into account the discrepancy between the source and target distributions of the data. When the density ratio (between the source and target distributions of the covariate) is unknown, a two steps pre-training procedure is carried out: the first step is devoted to the construction of a least squares SPDNN estimator of the density ratio; which is used in the second step to perform a pre-training reweighted SPDNN estimator of the regression function. For both the quantile and the Huber regression, non-asymptotic error bounds of the proposed SPDNN estimators are established in the class of H\"older smooth functions. These estimators can adaptively attain (up to a logarithmic factor) the minimax optimal convergence rate from i.i.d. data as well as from several classical time series models.
W. Kengne, Ehud Mossa Ockegna· arXiv.org· 0 citations
We present a simple Gaussian approximation to the finite-sample distribution of the classical ridge regression estimator. Our approximation captures the fact that, in finite samples, the ridge regression estimator trades off bias and variance to reduce estimation and prediction error. Our approximation is based on nonstandard asymptotics where $i)$ we let the estimator's regularization parameter grow proportionally to the sample size; and $ii)$ we treat the population regression coefficients as \emph{local} to the reference vector that defines the estimator's direction of shrinkage. In contrast to other asymptotic approximations in the literature, we allow for general forms of heteroskedasticity and autocorrelation in the data generating process (at the cost of considering a low-dimensional model where the number of covariates is not allowed to grow with the sample size). We use our simple Gaussian approximation to propose two new strategies to select the regularization parameter for the ridge regression estimator. The suggested strategies select the regularization parameter to minimize either average or worst-case excess prediction risk, where risk is computed using our suggested Gaussian approximation.
J. M. Olea, Ryan Strong, Amilcar Velez et al.· 0 citations
Cross-fitted estimators that transport information from the two labeled sources through source-specific density ratios are proposed that establish asymptotically linear inference for TPR and FPR, consistency and pointwise inference for the ROC curve, and asymptotically normal inference for AUC.
