Oct 2025· 2 citations· ⚡ 1 influential· 67 references
Computer Science
TL;DR
A single pre-training pipeline that builds transformer-based imputation specialists through three components: an entry-wise featurization that recasts imputation as supervised prediction over row--column context, a synthetic data generator with pluggable missingness modules, and prior-data fitting on millions of synthetic tables.
Abstract
Missing data in tabular datasets forces practitioners into a hard choice: deploy a general-purpose imputer that may perform poorly for the problem at hand, or wait for someone to design a specialized algorithm. This problem is worsened by the fact that real-world missingness rarely satisfies the textbook missing completely at random (MCAR) assumption, as entries are often missing not at random (MNAR), where the probability of being observed depends on the missing data itself. We collapse this trade-off into a single pre-training pipeline that builds transformer-based imputation specialists through three components: an entry-wise featurization that recasts imputation as supervised prediction over row--column context, a synthetic data generator with pluggable missingness modules, and prior-data fitting on millions of synthetic tables. Swapping only the missingness module, with no changes to architecture, loss, or training, yields a pattern-specific specialist that outperforms methods purpose-built for that pattern. We validate this on MissBench, a new benchmark of 42 OpenML datasets and 11 missingness patterns (including 9 MNAR variants) spanning medicine, finance, and engineering. Remarkably, training exclusively on MCAR yields a default model---TabImpute---robust across all tested patterns. In addition, the pattern-specific specialists produced by our pipeline outperform the 14 established baselines on their target patterns. We open-source the pipeline, models, and benchmark.
Predicting missing cell values in tabular data is a fundamental problem in data cleaning. While state-of-the-art reasoning models show great promise in predicting missing values in tables, by reasoning holistically across rows and columns, they are costly to deploy at scale and tend to be overconfident, often generating hallucinated or false-positive predictions. In this paper, we observe that achieving high-precision missing-value prediction in tables requires a distinct combination of three capabilities: (1) world knowledge, (2) text-based reasoning, and (3) code-based reasoning. We systematically explore design choices for combining these capabilities, and propose an Auto-Fill approach that post-trains three specialist small language models (SLMs), each optimized for one capability. We develop a calibrated ensemble mechanism that either dynamically selects the most confident specialist or abstains, ensuring high accuracy. Extensive experiments on 11 benchmarks with 2200 real tables drawn from diverse domains show that Auto-Fill achieves superior accuracy compared to state-of-the-art reasoning models (e.g., o3-pro, Gemini 3 Pro, and DeepSeek R1), while operating at a fraction (less than 1%) of the cost of these frontier models. Our results highlight the effectiveness of specialization and calibrated abstention in the important domain of tabular data. Auto-Fill is publicly available at https://github.com/lyrain2001/auto-fill.
Yurong Liu, Yeye He, Haoyu Dong et al.· 0 citations
Most machine-learning imputation techniques treat missing values independently of the downstream task, resulting in suboptimal predictive performance. While some recent methods jointly train an imputer with a target predictor, they fail to produce diverse and context-sensitive imputations and suffer from training inefficiencies. Additionally, in many methods, missing values are ignored during training, and the imputers are trained only on observed data. To overcome these limitations, we propose an imputation method, named end-to-end task-aware parallel Imputation with class-wise prototypes (ETPI). It captures the class-conditional distributions of input data using a few proxies. For the missing entries in any sample, ETPI generates class-aware pseudo-targets based on the predictor loss and the estimated class-conditional distribution. By exposing the imputer to both observed data and optimal pseudo-targets during training, ETPI effectively leverages training information to fit the imputer model and aligns imputation with the objectives of the target task. Extensive experiments on classification and regression tasks show that ETPI outperforms other state-of-the-art methods. It also maintains high imputation quality even with limited training data or high missing rates, mainly due to the high quality of the generated pseudo-targets and the integration of imputation and prediction into a single end-to-end pipeline.
Karrar Al-Kaabi, Davood Zabihzadeh· Engineering Research Express· 0 citations
Results support IAIML as a compact, interaction-aware framework appropriate for settings where bounded explanation size and controlled treatment of feature interactions are design requirements.
Large language models (LLMs) have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine learning workloads: predictive analytics over tabular data. This gap is the founding premise of the fast-growing field of tabular foundation models, but the question of why generic LLMs fail has remained open. We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning - and systematically evaluate five hypotheses for the failure: (a) an inability to handle noisy or non-linearly-separable data; (b) the linearised CSV format obscuring column structure; (c) the tokenisation of numeric values; (d) the number of test points classified per query; and (e) the dimensionality of the input. Controlled experiments falsify (a)-(d). Dimensionality, in contrast, is decisive: sweeping random linear projections of thirty-one benchmark datasets, the LLM is the only method among nine whose accuracy decreases as dimensionality grows, while every classical baseline stays flat or improves. A behavioural comparison against 252 configured classical models finds that in two dimensions the LLM predicts like a local, distance-based method (up to 91.6% grid agreement), but in higher dimensions no classical model - even when augmented with tuned, dimension-dependent noise - reproduces its predictions. We do not claim to have identified the internal mechanism; our results show, more modestly, that the LLM's capability dissolves with dimension in a way no noise-corrupted classical learner mimics - which explains why LLMs, so capable elsewhere, keep losing to fifty-year-old baselines on tables, while leaving the mechanism of the prediction as an open question.
Synthetic tabular data is increasingly used to support data sharing and machine learning development, where access to real data is restricted or sensitive. However, existing tabular data generators often require dataset-specific training or fine-tuning, data preprocessing, and extensive hyperparameter tuning to achieve plausible results. In addition, existing models may still struggle with mixed data types, modeling complex feature relationships, and pose a privacy risk for information leakage. With the rise of Large Language Models (LLMs) and their expanding capabilities across many domains, we investigate whether few-shot LLMs can be used in a training-free setting as tabular data generators. We propose FewShotTabLLM, a few-shot, training-free generation framework for tabular data that leverages schema-enriched context, representative sampling, and dataset profiling. We evaluate our approach across five widely used tabular benchmark datasets and compare it against strong training-based baselines from different model families (CTGAN, TVAE, TabDDPM, BeGReaT). A three-axis evaluation protocol has been used that covers representativeness, realism, and privacy. Across all evaluations, FewShotTabLLM outperforms baselines and surpasses them on many datasets, achieving realism, boundary adherence, and privacy preservation across all experiments. Overall, the results suggest that training-free fewshot LLM synthesis can serve as an alternative to existing models and a prominent research direction for improving dependency preservation and privacy-aware generation. Our code is available at: https://github.com/BMW-InnovationLab/FewShotTabLLM
Hadi Koubeissy, Michel El Khoury, Marc Kamradt et al.· Annual International Compute...· 0 citations
Structured data drives enterprise decision-making, yet building predictive pipelines for time-series and tabular modalities requires intensive feature engineering, model selection, and other ''tricks of the trade''. AutoGluon, an open source AutoML system, automates this through multi-layer stack ensembling, providing a unified API that reliably achieves state-of-the-art accuracy across time series and tabular data. Concurrently, Foundation Models for Structured Data (FM4SD) have emerged to push the boundaries of predictive performance, enabling both powerful zero-shot inference and efficient fine-tuning. Specifically, Chronos-2 delivers state-of-the-art multivariate forecasting for time series, while a rapidly growing ecosystem of tabular foundation models (e.g., TabPFN, Mitra, TabICL) transforms how practitioners execute classification and regression tasks. This system-focused tutorial teaches attendees how to operationalize FM4SDs using AutoML across temporal and tabular modalities, with an additional highlight on the newly released Chronos-2. We guide participants through a technical progression: establishing AutoGluon as the orchestration framework, detailing the internal mechanics of Chronos-2 and various tabular FMs, and unifying them into production-ready pipelines.
Boran Han, Nick Erickson, Abdul Fatir Ansari et al.· Proceedings of the 32nd ACM...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 18, 2026
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
