Predicting missing cell values in tabular data is a fundamental problem in data cleaning. While state-of-the-art reasoning models show great promise in predicting missing values in tables, by reasoning holistically across rows and columns, they are costly to deploy at scale and tend to be overconfident, often generating hallucinated or false-positive predictions. In this paper, we observe that achieving high-precision missing-value prediction in tables requires a distinct combination of three capabilities: (1) world knowledge, (2) text-based reasoning, and (3) code-based reasoning. We systematically explore design choices for combining these capabilities, and propose an Auto-Fill approach that post-trains three specialist small language models (SLMs), each optimized for one capability. We develop a calibrated ensemble mechanism that either dynamically selects the most confident specialist or abstains, ensuring high accuracy. Extensive experiments on 11 benchmarks with 2200 real tables drawn from diverse domains show that Auto-Fill achieves superior accuracy compared to state-of-the-art reasoning models (e.g., o3-pro, Gemini 3 Pro, and DeepSeek R1), while operating at a fraction (less than 1%) of the cost of these frontier models. Our results highlight the effectiveness of specialization and calibrated abstention in the important domain of tabular data. Auto-Fill is publicly available at https://github.com/lyrain2001/auto-fill.
The results show that the LLM's capability dissolves with dimension in a way no noise-corrupted classical learner mimics - which explains why LLMs, so capable elsewhere, keep losing to fifty-year-old baselines on tables, while leaving the mechanism of the prediction as an open question.
Supervised fine-tuning is often discussed in terms of data volume and target quality, but the format of the supervision itself can change the reasoning strategy a model expresses in its outputs. We study this effect on a six-category reasoning benchmark spanning symbol transformation, text decryption, bit manipulation, gravitational constant estimation, numeral conversion, and unit conversion (drawn from the NVIDIA Nemotron Model Reasoning Challenge). Using NVIDIA Nemotron-3-Nano-30B-A3B with matched LoRA training settings, we compare three symbol-supervision formats: verbose English rule descriptions, compact family tags, and compact formula notation. We hypothesize that supervision renderings bias token-level reasoning priors, and that these priors transfer across task boundaries in multi-task SFT. In the canonical strict-rescore inventory, the best compact tag and formula checkpoints are statistically equivalent in aggregate within a pre-specified ±4-point margin: K8A-800 reaches 72.3% strict-scored overall accuracy and K8B-700 reaches 71.2% (TOST p = 0.003). Compact tags nevertheless provide a cleaner behavioral profile: an earlier K8A-400 checkpoint reaches 66.4% overall, 98.7% gravity accuracy, and 36.9% bit accuracy without the same contamination signatures. In contrast, verbose English rule descriptions are associated with heuristic parroting, with up to 57% of symbol failures at audited verbose checkpoints collapsing to a single remove-operator template, while formula notation is associated with cross-category contamination: numeric-looking predictions appear more often in text decryption (higher at five of six matched training steps under the canonical seed; matched-step means 15.8 vs. 11.7 numeric predictions per 157 text rows), and gravity failures at a representative K8B formula checkpoint shift toward shortcut stubs and explicit g = 9.8/9.81 fallbacks. We further show that checkpoint selection and strict evaluation auditing materially change branch decisions. Across three training seeds, neither compact format shows a consistent aggregate advantage, while the contamination signatures are partly seed-specific: the gravity-shortcut severity difference persists but is not exclusive to the formula branch, and the numeric–text signature does not reproduce under reseeding. These results support treating supervision format as a first-class hyperparameter for multi-task reasoning SFT in large language models—at least in this benchmark-and-model setting—rather than a mere rendering detail. Because such symbolic and procedural reasoning tasks recur in domains including cybersecurity, mathematics, and code generation, the same formatting choices plausibly shape the policy that any later reinforcement-learning stage would inherit, which we flag as future work.
Nhat Thanh Vu, M. Rashid, Fariza Sabrina· Electronics· 0 citations
Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly. In these settings, accuracy alone is insufficient: models must also know when they are likely to be wrong. We present ConfidenceBench, a calibration benchmark that evaluates verbalized confidence estimates in 15 frontier LLMs using the Brier score, a proper scoring rule that incentivises truthful probability reporting. Confidence is elicited via prompting, requiring no access to model logits and making the framework applicable to both closed-source and open-source systems. The benchmark comprises 200 private multiple-choice questions across four categories: spatial reasoning, high-precision mathematics, word lookup, and unknowable questions. Across three independent runs, Claude Opus 4.6 and Gemini 3.1 Pro Preview achieve the lowest Brier scores, both reported as 0.103. Both substantially outperform the calibrated-random baseline of 0.1875, while Gemini 3.1 Flash-Lite scores 0.367, indicating severe miscalibration. Accuracy and calibration diverge substantially across model families: the most accurate model is not the best-calibrated, and several models perform worse than the calibrated-random Brier baseline despite reasonable accuracy. These results show that verbalized confidence calibration is a distinct and practically important axis of LLM reliability, complementary to standard accuracy-based evaluation.
M. ffrench-Constant, Daniel Yang, Xinmeng Huang et al.· 0 citations
Reliable confidence estimation is essential for deploying large language models (LLMs) in confidence-aware systems, where downstream decisions such as retrieval, tool use, and adaptive computation depend on accurately estimating answer reliability. Existing approaches, however, largely treat confidence as a property of completed responses, overlooking how confidence-related information evolves throughout the answering process. In this work, we investigate confidence from a temporal perspective by comparing pre-solution Feeling-of-Knowing (FOK) and post-solution Judgement-of-Learning (JOL) confidence estimates across frontier and open-source LLMs. We show that post-solution confidence is consistently better calibrated and more discriminative than pre-solution confidence, while linear probes trained on hidden representations recover substantially richer confidence-related information than models explicitly verbalise. Building on this observation, we introduce future confidence distillation, which trains predictors operating on pre-solution hidden representations using teacher confidence estimates produced by post-solution correctness probes. Despite requiring only pre-solution representations for inference, distilled predictors recover much of the calibration improvement achieved by post-solution confidence, remain highly sample efficient, and transfer across datasets within the same domain. Together, our findings demonstrate that confidence-related information evolves throughout the answering process and can be anticipated before answer generation is complete, enabling significantly more reliable yet low-cost confidence estimation.
Large language models (LLMs) are increasingly deployed in high-stakes settings, yet reliably estimating when their outputs should be trusted remains an open challenge. Existing uncertainty estimation approaches—such as calibration, token-level probabilities, or semantic entropy—typically require access to model internals, additional supervision, or computationally intensive pipelines. We propose answer instability, defined as the variability of a model’s final answer across repeated stochastic generations of the same prompt, as a simple, label-free, and black-box uncertainty signal. Evaluated across three task types — reasoning, multiple-choice QA, and constraint-following — using four LLMs and 520 prompt-model pairs, our approach achieves performance competitive with semantic entropy while requiring no semantic similarity model. Our results show that instability strongly correlates with prediction errors and reliably discriminates correct from incorrect outputs. We further demonstrate its utility for selective prediction and targeted repair, improving reliability without access to internal probabilities or additional training.
J. M. Araña, Kristine Ann M. Carandang, E. R. Casin et al.· Annual Meeting of the Associ...· 0 citations
Tabular Foundation Models (TFMs) increasingly rely on in-context learning, where a model receives labelled examples at inference time and predicts labels for new inputs without updating its weights. Existing TFMs are typically trained on either massive synthetic corpora or very large collections of real datasets. In contrast, we show that surprisingly strong transfer can emerge from self-supervised pre-training on just a single real table. In this setting, we also find that tables tend to be either broadly useful or broadly poor regardless of downstream prediction task, and that the strongest predictor of usefulness is the number of features rather than the number of instances. This leads to a task-centric interpretation of tabular pre-training: the number and the quality of tasks are essential for the pre-training of TFMs. We show that the same task-centric perspective can help corpus design at scale: fine-grained column-level pre-processing consistently improves downstream performance, while no improvements are observed when we filter or deduplicate at the dataset level. Finally, we offer a new perspective for how TFMs generalize: we believe that tabular in-context generalization is largely retrieval-based, and good models are those that learn to identify relevant examples in the provided context and aggregate them well. The mechanics of TFMs have been relatively understudied; our task-centric, retrieval-based perspective offers a new framework to guide future model and corpus design.
Nour Shaheen, Junwei Ma, Alex Labach et al.· 0 citations