Skip to content

Multi-Task Learning with Covariate-Overlap Regularization

May 2025 · 0 citations · 27 references
Mathematics Computer Science

TL;DR

COVariate-ovERlap regularized multi-task learning (COVER) is proposed to address covariate and posterior heterogeneity and achieves the lowest response-averaged prediction error among the compared methods and reveals tissue-pair integration patterns.

Abstract

Multi-task learning improves data efficiency by sharing information across related tasks, but indiscriminate sharing can be harmful when their covariate distributions and response relationships differ. We propose COVariate-ovERlap regularized multi-task learning (COVER) to address covariate and posterior heterogeneity. The model combines a common component function with a shared neural representation and low-dimensional task-specific coefficients. Taskwise second-moment matrices summarize covariate heterogeneity and determine the strength of coefficient integration in each representation direction. We derive a covariate-overlap penalty by minimizing the total squared change in two task predictors when their coefficients are replaced by one auxiliary coefficient. An equivalent auxiliary formulation supports end-to-end training without matrix inversion. An exact fixed-representation bias--variance decomposition quantifies how covariate overlap controls variance reduction and how posterior heterogeneity determines shrinkage bias. Global and localized end-to-end oracle inequalities account for jointly learning the neural functions and estimating the overlap matrices from the same observations. We give explicit neural-network rates and sharpen the stochastic prediction term when the regularized oracle risk and overlap-estimation error are small. Simulations across diverse heterogeneity settings show competitive performance against deep-learning and statistical data-integration methods, with the largest gains under joint heterogeneity. In a GTEx central-nervous-system analysis, COVER achieves the lowest response-averaged prediction error among the compared methods and reveals tissue-pair integration patterns.

View source

Similar papers

Preprint Sep 2026

Joint-Sparse Transfer Learning for High-Dimensional Multi-Output Regression

Multitask linear models can improve estimation and prediction by exploiting structure shared across responses, bridging taskwise fitting and complete pooling. In many applications, however, the objective is estimation in a data-limited target domain, while data-rich but heterogeneous source domains are available. Borro...

Sun-Woo Lim, Mladen Kolar · 0 citations
#machine learning Preprint Oct 2026

Latent-Space Importance Weighting under Covariate Shift

In this work, we propose a transfer learning algorithm for covariate shift, integrating sample reweighting with representation learning. A key feature of our approach is the use of a lower-dimensional latent space for modeling, in contrast to conventional methods that operate directly in the high-dimensional ambient sp...

Yu-Ling Jiao, Li-Can Kang, Yang Xiang et al. · 0 citations
Preprint Oct 2026

Empirical-Bayes spectral partial pooling across related tasks

Spectral methods are central to high-dimensional statistics and machine learning, underlying procedures for covariance estimation, matrix denoising, representation learning, clustering, and latent variable modeling. In this work, we focus primarily on spectral estimators for high-dimensional factor models, where leadin...

L. Mauri · 0 citations
Book Open access Nov 2026

Multi-Task Dense Prediction with Mixture of Unit-Rank Experts

Multi-task dense prediction has recently achieved significant performance gains but remains challenging in learning fine-grained and discriminative task-specific representations. In this paper, we propose a novel decoder-focused Mixture of Unit-Rank Experts (MURE) for multi-task dense prediction. Inspired by the subspa...

Hao Xu, Zhen-Chun Wei, Zheng Wang et al. · 0 citations

Multitask Regression with Pairwise Fusion

We study multitask regression when coefficient sharing can differ by predictor. For a given predictor, many tasks may have the same coefficient while a few differ, and the exceptional tasks need not be the same for another predictor. We describe this structure by two quantities: the number of active predictors and the...

Xiao-Dong Li, Zhen-Tao Li · 0 citations

Sample-Weighted End-to-End Trace-Norm Geometry for Multitask Learning

The sample-size-weighted trace norm of the end-to-end map from task coefficients to input-space predictors is studied instead the sample-size-weighted trace norm of the end-to-end map from task coefficients to input-space predictors.

Mahdi Mohammadigohari · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.