The proposed TabLoRA is a parameter-efficient trainable neural ensemble for large-scale tabular learning that improves the feasibility of neural ensemble learning while preserving much of the benefit of full ensembles.
Abstract
Tabular learning is still dominated by gradient-boosted decision trees (GBDTs), while recent deep learning approaches have become increasingly competitive. However, applying deep tabular models to large-scale datasets remains challenging, as large sample sizes, high feature dimensionality, or many target classes can introduce substantial computational cost. We propose TabLoRA, a parameter-efficient trainable neural ensemble for large-scale tabular learning. Instead of using fully independent ensemble backbones, TabLoRA shares a common backbone across predictors and introduces predictor-specific low-rank adaptations, enabling ensemble-style prediction without full parameter duplication. Across benchmarks, TabLoRA achieves a favorable balance between predictive performance and practical efficiency compared with GBDT methods and recent deep learning baselines under the same resource constraints. Memory analysis and ablation studies further show that the proposed design improves the feasibility of neural ensemble learning while preserving much of the benefit of full ensembles.
Experiments on the million-row HIGGS and SUSY datasets show that 512 prototypes retain strong predictive performance and reliable calibration, corresponding to an approximately 1,953-fold context compression.
Mahboobe Jadid, Melika Rezaye Garkani, A. Mousavi· 0 citations
TabNSM provides an effective and scalable approach to deep tabular regression, and demonstrates that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.
In deep learning for tabular data, efficient ensembles of multilayer perceptrons (MLPs) have recently emerged as effective and practical architectures. Existing methods of this kind use the same hyperparameters for all underlying MLPs, which requires hyperparameter tuning for achieving the best performance. In this work, we introduce TabPack, an efficient MLP ensemble with strong out-of-the-box performance and reduced reliance on traditional tuning. In a single run, TabPack samples and trains many MLPs with different hyperparameters efficiently in parallel and selects ensemble members on the fly during training. Thus, TabPack only requires specifying ranges from which to sample MLP hyperparameter rather than exact hyperparameter values, which naturally demands less precision for good performance. In experiments on medium-to-large public datasets, TabPack with default settings performs on par with extensively tuned prior methods, thus substantially reducing effort and compute resources needed to achieve competitive results on tabular tasks. Notably, running the default TabPack configuration on a modern MacBook took less time than tuning some baselines on an industry-grade GPU.
Yury Gorishniy, Akim Kotelnikov, Ivan Rubachev et al.· 0 citations
Localized TabICLv2 introduces a method that reduces the inference cost of TabICLv2 by retrieving only the k nearest training neighbours for each test point, measured by similarity in the model's Stage 2 row-representation space, rather than using the full training context.
The advent of artificial intelligence has significantly enhanced credit‐scoring technologies. Despite the remarkable efficacy of advanced deep learning models, mainstream adoption continues to favor tree‐structured models due to their robust predictive performance on tabular data. Although pretrained models have seen considerable development, the use of such models for tabular‐structured credit‐scoring datasets remains largely unexplored. Tabular‐oriented large models, such as TabPFN, have made the application of large models in credit‐scoring feasible, albeit they can only process limited sample sizes. This paper proposes a novel framework that combines a tabular‐tailored dataset distillation technique with a pretrained model, thereby improving the scalability of TabPFN. Furthermore, although class imbalance is a common characteristic of financial datasets, its influence during dataset distillation has not been systematically explored; we therefore integrate imbalance‐aware objectives into the distillation process, resulting in improved performance on imbalanced credit data. Experiments on six publicly available credit‐scoring datasets show that the proposed imbalance‐aware distillation improves AUC by up to 8.7 percentage points over the standard MSE‐based distillation baseline. At the same time, distilled datasets retain roughly 76%–95% of the full‐data AUC while often using under 10% of the original training samples (and at most 31.3%) and consistently outperforming random subsets. Taken together, these results demonstrate that imbalance‐aware dataset distillation provides a compact yet accurate solution for imbalanced credit scoring and enhances the practical scalability of tabular pretrained models such as TabPFN.
Xia Li, Hanghang Zheng, Xiwei Zhuang et al.· Intelligent Systems in Accou...· 0 citations
REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation, and a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters.
Peng Liu, Hui-Bing Zeng, Yi-Qun Zhang et al.· IEEE Transactions on Emergin...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
