Generalized context in cross attention for transfer learning of disjoint tabular data.
TL;DR
The experiments show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models.
Abstract
Unlike images and text, transfer learning on tabular data is challenging due to heterogeneity in feature types, structures, and semantics across disparate domains. Existing methods assume shared features across data tables to enable knowledge transfer between domains, which is unrealistic in practice. This paper introduces generalized context learning to remove the requirement of shared features across domains. The generalized context captured by transformer projection weights for key, value, and query provides rule-based generalization rather than the domain-specific context conventionally learned from transformer activations. Projection weights for key from the source domain interact with the weight for query in the target domain to achieve Cross-domain Attention Transfer Learning (CATTLE) in a data-agnostic manner. Our experiments on ten pairs of disjoint source-target data sets show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models. CATTLE achieves the best average rank (2.9) and delivers a 3.7% average AUROC gain over the baseline methods. The CATTLE source code is available at https://anonymous.4open.science/r/cattle-ED9F.