Learning Operators by Regularized Stochastic Gradient Descent with Operator-valued Kernels

Abstract

We consider a class of statistical inverse problems involving the estimation of a regression operator from a Polish space to a separable Hilbert space, where the target lies in a vector-valued reproducing kernel Hilbert space induced by an operator-valued kernel. To address the associated ill-posedness, we analyze regularized stochastic gradient descent (SGD) algorithms in both online and finite-horizon settings. The former uses polynomially decaying step sizes and regularization parameters, while the latter adopts fixed values. Under suitable structural and distributional assumptions, we establish prediction and estimation error bounds with no explicit dependence on the dimension of the output space. The resulting convergence rates are near-optimal in expectation, and we also derive high-probability estimates that imply almost sure convergence. Our analysis introduces a general technique for obtaining high-probability guarantees in infinite-dimensional settings. We illustrate the scope of our framework through applications to structured prediction and a class of parametric elliptic PDEs. For the latter, we construct kernels for arcsine and uniform sampling, verify the assumptions of our high-probability results, and obtain bounds uniform over finite parameter truncations under suitable conditions on coefficient decay.