DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining

Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training. We propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target. Specifically, we define an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run, and estimate this quantity from intermediate checkpoints without retraining. Applying the method to 18 configurations from the Pythia and PolyPythia suites, we find systematic temporal changes in influential data. Early in training, literature-related data are more strongly aligned with the trajectory toward the final parameters, whereas STEM data become more strongly aligned in later stages. This qualitative crossover is broadly consistent across model configurations. Our results provide a tractable trajectory-level view of how influential data change throughout pretraining, complementing influence analyses defined with respect to specific downstream tasks or validation sets.

LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations

Language models (LMs) increasingly drive real-world applications that require world knowledge. However, the internal processes through which models turn data into representations of knowledge and beliefs about the world are poorly understood. To facilitate such studies, we present LMEnt, a suite including (1) a knowledge-rich pretraining corpus, fully annotated with entity mentions based on Wikipedia, (2) an entity-based retrieval method over pretraining data that outperforms existing tools by as much as 80.4%, and (3) 12 pretrained LMs with up to 1B parameters and 4K intermediate checkpoints, with comparable performance to popular open-source models on knowledge tasks. Together, these resources provide a controlled environment for analyzing connections between entity mentions in pretraining data and downstream performance. We show the utility of LMEnt by studying knowledge acquisition over training, finding that entity co-occurrence and mention forms—which are difficult to study with existing tools—affect learning trends. Moreover, as LMs form stronger associations between entities, their facts are harder to edit in-context, whereas inconsistencies in model predictions over training are indicative of editing success. We release LMEnt to support studies of knowledge in LMs, including knowledge representations, plasticity, editing, attribution, hallucinations, and learning dynamics. huggingface.co/LMEnt github.com/LMEnt

OmniOData: Unleashing Small Language Models for OData Query Generation with Synthetic Data and Reinforcement Learning

Despite the success of Large Language Models (LLMs) in structured query generation, OData—a critical RESTful protocol for enterprise APIs—remains under-researched due to a lack of high-fidelity, execution-validated datasets. To bridge this gap, we introduce O M - NI OD ATA , a framework that generates S YN O-D ATA , the first large-scale OData corpus featuring execution-grounded queries and reasoning traces. Using this corpus, we develop O MNI OD ATA -R1 (1.5B–3B parameters), a family of models that match or surpass frontier proprietary systems, such as GPT-4o and Gemini 3, on realistic industrial benchmarks. Our results demonstrate that the synergy of execution-verified synthetic data and Reinforcement Learning (RL) effectively unlocks the latent reasoning of Small Language Models (SLMs), providing a high-performance, low-latency solution for specialized enterprise query generation. The code and data will be released under an open-source license.

Beyond Language Processing: LLMs Rules-Injected Instruction Tuning for Traffic Prediction

Large Language Models (LLMs) demonstrate strong capabilities in contextual integration and multi-step reasoning, which endow them with the potential to model heterogeneous traffic data. However, the knowledge acquired during LLM pre-training is primarily qualitative and broad, and does not provide the fine-grained, context-specific quantitative dependencies required for accurate traffic prediction. This mismatch between pre-training knowledge and task requirements can reduce the effectiveness of instruction tuning, the standard approach for adapting general-purpose LLMs to downstream tasks. To address this issue, we propose Rules-Injected Instruction Tuning (RuleGPT), a paradigm that injects IF–THEN rules extracted from training data directly into the LLM's instruction sequence. For each sample, relevant rules are retrieved and encoded as learnable rule tokens that are concatenated with the input, providing explicit guidance to the model. By encoding low-order dependencies in this way, RuleGPT allows the LLM to focus on higher-order, context-dependent interactions. Experiments on real-world traffic datasets demonstrate that RuleGPT consistently improves prediction performance over standard instruction tuning and strong forecasting baselines. The code is available at https://github.com/tigerbunny2023/RuleGPT.

Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT

A controlled experiment on the final window of pretraining, the last data trained on before instruction tuning, finds that what a model is pretrained on last shapes how it reacts to alignment, and what it was trained on last should be reported with it.

SciPedia: Unlocking the Value of Scientific Data for Pre-training