Few-Shot LLMs as Synthetic Tabular Data Generators

PFN-Syn: Generating Synthetic Tabular Data with Prior-Data Fitted Networks

TabClean: Scalable Tabular Data Cleaning via Reusable LLM-Synthesized Programs

LAB-Tab: LLM-Augmented Bayesian Network Adaptation for Few-Shot Tabular Generation

Tabular data generation supports analysis and decision-making when target-domain data are scarce, yet collecting complete target samples is often costly. A practical but underexplored setting provides only a few target records together with richer source data from a related domain. Existing few-shot tabular generators often either fit sparse target statistics directly, which can overfit incidental patterns, or reuse source-domain generators, which may preserve dependencies that no longer hold in the target domain. To address this problem, we propose LAB-Tab, an LLM-augmented Bayesian network (BN) adaptation framework for source-aware few-shot tabular generation. LAB-Tab first fits a BN from source data and then uses an LLM to propose plausible target-domain BN edges that are absent from the source BN graph. This step converts semantic and weak statistical evidence into explicit structural hypotheses, thereby expanding the editable edge space beyond the source-fitted graph. Because the proposed edges may be noisy and interact with existing dependencies, a PPO policy calibrates edges in the augmented BN through edge-level actions, including keep, weaken, strengthen, flip, and deactivate. The PPO policy is trained with a reward that combines distributional alignment, downstream utility, and preservation of target-relevant dependencies. The adapted BN is then sampled to synthesize target-domain tables. Across six source--target distribution-shift scenarios built from three US Census (ACS) prediction tasks, LAB-Tab achieves the best performance at the 10% target-data budget, leads four of the six individual scenarios, and reduces the macro Overall score by 33.8% relative to the strongest baseline. It also obtains the best macro JSD, WAPE, and UtilityGap while maintaining competitive feature--label preservation.

Data-Efficient Adaptation of LLMs via Attention Head Reweighting

Learning effectively from limited data is critical in domains like security where labeled examples are scarce. Large language models (LLMs) have demonstrated some capabilities for data-efficient learning, especially through parameter-efficient adaptation methods, but continue to struggle when faced with few samples for difficult tasks. To meet this challenge, we propose Attention Head Reweighting (AHR), a data-efficient method that adapts LLMs to new text-classification tasks by learning only a single scalar per attention head. This drastically reduces the number of parameters that need to be learned by making use of the functional specialization of individual attention heads. Experiments on diverse open-source text classification datasets show that AHR can outperform standard baselines like LoRA when learning from limited samples, despite having 200-1000x fewer trainable parameters, as our AHR only modifies ~0.0001% of the model's parameters. In addition, our learned weights are easy to interpret and can be analyzed to better understand the mechanisms and attention heads responsible for in-context learning abilities in LLMs.

Learning on Imbalanced Noisy Data via Debiased Sample Selection and LLM-Driven Annotation

Łearning with Noisy Labels (LNL) is a challenge that arises in many scenarios where training data can contain noisy labels. While various methods, such as active learning for small language models (SLMs), have been proposed to re-annotate samples, they still require human efforts. The prevalent in-context learning (ICL) of large language models (LLMs) can perform text annotation tasks, but their efficiency hinges on the precise selection of clean versus noisy samples from noisy data. Meanwhile, this paper observes that existing sample selection suffers from selection and training bias on class imbalance noisy datasets, leading to decreased accuracy in sample selection. To bridge it, we pro-pose a debiased sample selection and an innovative collaborative learning framework DeCo based on active learning for LNL. During collaborative learning, we first conduct debiased sample selection by designing a robust expert model based on SLMs and introducing a dynamic class-wise threshold strategy, then feed selected clean samples to active annotator LLMs for re-annotating noisy samples us-ing ICL, with the re-annotation results reinforcing SLMs’ training for subsequent accurate selection. Ultimately, we employ distinct loss functions adept at managing subsets with varying degrees of label noise. Extensive experimental results on synthetic and real-world datasets demonstrate the effectiveness and superiority of our proposed method.

Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning

Model collapse is a central challenge in learning from synthetic data: as later-generation large language models (LLMs) are trained on an increasing proportion of model-generated data, performance can degrade due to narrowed coverage and accumulated bias. Existing work mainly studies how to bound this degradation. In iterative model evolution, however, the more meaningful objective is to ensure that each successive model improves over its predecessor, which requires diagnosing collapse at a granularity that is actionable for data curation. We study this problem in synthetic data self-improving for instruction tuning. We show that collapse in this setting is not simply uniform performance degradation, but can appear as a polarization of competence, where synthetic training reinforces already strong skills while further degrading weak ones. Motivated by this observation, we propose KITE (Knowledge-boundary Instruction Tuning via Exploration), a two-stage framework that combines failure-guided data generation with boundary-aware uncertainty curation. Experiments across several datasets and multiple open-source LLMs show that KITE yields more stable improvement than strong synthetic-data baselines.