A (computationally inefficient) adaptive estimator that, so long as $p$ is a mixture of $k$ symmetric log-concave densities, achieves error comparable with the optimal estimator that knows $p$ and has $\tilde\Theta(n/k)$ samples.
Abstract
We study finite-sample linear regression in the presence of varied and unknown label noise, focusing on the heteroskedastic and adaptive linear regression models. Heteroskedastic linear regression models settings where the labels are of varying quality. We receive $n$ pairs $(X_i,Y_i)$ with labels $Y_i=X_i^\top\beta+\varepsilon_i$, where $\varepsilon_i\sim N(0,\sigma_i^2)$ and the variances are unknown to the estimator. One natural measurement of the difficulty of this problem is the number of samples $m$ for which $\sigma_i^2\le1$ (larger $m$ is easier). We obtain a polynomial-time estimator with rate $\tilde{O}((nd^3/m^4)^{1/6})$ when $m\gg d^{3/4}n^{1/4}$, as well as nearly-matching lower bounds. For $d=O(1)$, our estimator achieves error $o(1)$ when $m\gg n^{1/4}$, whereas $L_1$ regression and other traditional approaches require $m\gg n^{1/2}$. In adaptive linear regression, the errors are drawn i.i.d. from an unknown distribution $p$, and our goal is to design a generic estimator that performs nearly as well as the best custom estimator that knows $p$. We introduce a (computationally inefficient) adaptive estimator that, so long as $p$ is a mixture of $k$ symmetric log-concave densities, achieves error comparable with the optimal estimator that knows $p$ and has $\tilde\Theta(n/k)$ samples. For $k=1$, we show that $L_q$ regression (with data-dependent $q$) gives a polynomial-time estimator. Finally, to study the computational limits of both problems, we introduce the planted linear regression problem, where $X_i\sim N(0,I_d)$, $m$ unknown samples are noiseless, and the rest have error $\varepsilon_i\sim N(0,1)$. We conjecture that recovering $\beta$ up to error $\ll\sqrt{d/n}$ (or exactly) may have an information-computation gap between $m=d+1$ and $m\sim d^{3/4}n^{1/4}$, as is suggested by our near-matching polynomial-time estimator and statistical query (SQ) lower bound.
We study estimation of a constant conditional variance $\sigma^2$ in nonparametric regression with a $d$-dimensional random design. This is an important problem, and similar questions arise in causal inference. The regression function is $\beta_b$-H\"older smooth, the design density is $\beta_g$-H\"older smooth and bounded above and away from zero, and we consider the nonparametric regime $\beta_b>1$ and $d>4\beta_b$. Set $\beta_g^\star=\beta_b(1-4\beta_b/d)/\{1+2\beta_b/d+8(\beta_b/d)^2\}$. We give an estimator whose mean squared error is upper bounded by $Cn^{-4(\beta_b+1)/(d+4)}$ in the low-regularity regime when $0<\beta_g\leq\beta_g^\star$. The low-regularity branch is based on a new two-scale construction: the covariate space is partitioned into cells, the local polynomial trend is projected out within each suitable cell, and the squared normalized contrast from one eligible close pair per cell is averaged across cells. In the high regularity regime when $\beta_g>\beta_g^\star$, a higher-order influence function estimator of Robins, Li, Tchetgen Tchetgen, and van der Vaart (2008) provides the rate $Cn^{-8\beta_b/(d+4\beta_b)}$. We also give an all-pairs ridge extension, which achieves the same two-scale rate, and evaluate the methods alongside a range of existing estimators in simulations.
Edgar Dobriban, Rajarshi Mukherjee, James M. Robins et al.· 0 citations
Consider the partial linear model $Y = \mu_0(X) + \beta_0 \cdot T + \varepsilon$ and $T = \pi_0(X) + u$ in the structure-agnostic setting, where we are blind to the structure $\mu_0$ and $\pi_0$ and estimate the nuisances by a black-box hypothesis class. The learnability of the class is characterized by the estimation error $\delta_s$ in the absence of model misspecification and its $L_2$ mis-specification error $\delta_{a, \mu}$ and $\delta_{a, \pi}$ for $\mu_0$ and $\pi_0$, respectively. We propose a novel estimator of the target linear coefficient $\theta_0 = \beta_0$ with error rate \[ \frac{1}{\sqrt{n}} + \delta_{a, \mu} \cdot \delta_{a, \pi} + [\delta_s]^2. \] A matching lower bound is also established, implying that this rate is unimprovable. Compared with the product rate yielded by double machine learning (DML), our estimator removes the suboptimal term $\max(\delta_{a, \mu}, \delta_{a, \pi})\cdot \delta_s$ at no extra cost or assumption. Building on the underlying insights, which are neither tailored to the one-learner setting nor the partial linear model, we propose Transductive Adversarial Moment-calibrated Editing (TAME), which locally edits debiasing weights induced by black-box regression estimates on the inference sample through adversarial conditional moment calibration. TAME can be combined with any initial black-box estimates and can strictly improve on DML guarantees when the nuisance difficulties are imbalanced. We discuss how to fully exploit the advantages introduced by TAME, including the gains from using two learners, the resulting under-smoothing principle for model selection, and extensions to other linear functional estimation problems.
Given $[0,1]$-valued random variables $X_1,\dots,X_n$ such that $\mathbb{E}[X_i | X_1,\dots,X_{i-1}]= \mu$ for all $i$, we propose a new nonasymptotic confidence interval for $\mu$ that is obtained by inverting terminal e-values generated by a novel betting strategy. When the data are iid, its limiting width matches that of the central limit theorem (``Gaussian-efficient''), finally surpassing the inefficient limits of previous betting intervals. Our main conceptual advance involves designing betting fractions that track the conditional rejection probability of the most powerful terminal test in a limiting Gaussian experiment. When one predictable variance estimator is shared across candidate means, the deterministic inversion is an interval for every data sequence and its two endpoints can be found easily. The width can be improved further with external randomization. In simulations, our method yields the tightest intervals to date; for every distribution tested and all sufficiently large $n$, our deterministic version beats STaR-Bets and is competitive with Gaffke, while the randomized improvement beats both. It thus combines finite-sample validity under martingale dependence, easy endpoint computation, Gaussian-efficient inference for iid data, and excellent empirical performance. We also extend the construction and its efficiency theory to sampling without replacement, where it again achieves state-of-the-art empirical performance.
Diego Martinez-Taboada, Aaditya Ramdas· 0 citations
Let $\boldsymbol{X}_i=(X_{1i},X_{2i})^\top$ be independent and identically distributed observations with mean $\boldsymbol{\mu}=(\mu_1,\mu_2)^\top$ constrained by $\mu_1\leq\mu_2$. We study empirical-likelihood inference for a fixed mean vector and distinguish it from the previously known test of equality against an ordered alternative. At a fixed interior point, the constrained empirical likelihood ratio has the usual $\chi^2_2$ limit. At a fixed boundary point $(m,m)^\top$, its limit is the chi-bar-square distribution $\tfrac12\chi^2_1+\tfrac12\chi^2_2$. By contrast, profiling the unknown common mean in the equality-versus-order test yields $\tfrac12\chi^2_0+\tfrac12\chi^2_1$, the $k=2$ ordered-mean case of El Barmi (1996). We give an exact reduction of the latter statistic to the empirical likelihood of the paired differences, establish the localization step needed for the fixed-boundary expansion, and derive a local-to-boundary limit showing that interior calibration is not uniform over $n^{-1/2}$-neighborhoods of the boundary. Monte Carlo experiments under Gaussian, Student $t_5$, and shifted log-normal sampling examine fixed, boundary, and local regimes with explicit numerical-failure accounting. Illustrative paired-data analyses show the practical distinction between fixed-candidate confidence regions, directional equality tests, and ordinary scalar empirical-likelihood intervals truncated to the nonnegative parameter space.
Under mild design conditions, satisfied by a broad class of correlated random designs, it is shown that EBMoM consistently estimates a growing number of moments and hence the prior itself, provided that $n\geq p^{1-o(1)}$.
Zhou Fan, Yandi Shen, Hao-Yu Wang et al.· 0 citations
The power prior of Ibrahim and Chen incorporates historical data into a Bayesian analysis by raising the historical likelihood to a power $a_0 \in [0, 1]$. The choice of the exponent has remained an open question. This paper gives a closed-form answer under the predictive log-loss. For a model with $d$ parameters, a historical sample of size $N_0$, and average Kullback--Leibler divergence $\bar{D}_0$ between the historical and current data-generating distributions, the optimal exponent is $a_0^{*} = d/(2 N_0 \bar{D}_0 + d)$. Equivalently, the optimally borrowed effective sample size obeys the harmonic law $1/E^{*} = 1/N_0 + 2\bar{D}_0/d$: compatible data are pooled in full, and any difference caps the borrowed information at $d/(2\bar{D}_0)$ observations. The result is exact for multinomial data and extends to smooth parametric families. The law benchmarks adaptive borrowing, explains the reported degeneracy of the normalized power prior, and shows that neither subsetting the data nor decaying the exponent improves on the correctly discounted constant.
Yuriy A. Reznik· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
