SciPedia: Unlocking the Value of Scientiﬁc Data for Pre-training

Yiwei Qin Zhen Huang Tiantian Mi Weiye Si Qipeng Guo Siyuan Feng Pengfei Liu Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot Ashish Sabharwal Carissa Schoenick Oyvind Tafjord. 2018 K. Cobbe Vineet Kosaraju Mo Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Gheorghe Comanici E. Bieber Mike Schaekermann

· 0 citations · 50 references

Semantic Scholar

View source

Similar papers

Preprint Jul 2026

Index SLM Technical Report

Control studies on model depth, learning-rate magnitude and scheduling, the interaction between learning-rate decay and data quality, and the effect of including instruction data during pre-training are reported, and an unexplained surge in benchmark performance midway through the constant-learning-rate phase is docume...

Lusheng Zhang, Shien He, Tianxing Yan et al. · 0 citations

Jun 2026

NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research

This report provides a fully reproducible empirical post-training recipe for 8B-scale LLMs, and comprehensively dissects the capability trade-offs among instruction adherence, mathematical reasoning, code generation and general knowledge.

Qiaobo Hao, Yangqian Wu, Shunyi Wang et al. · arXiv.org · 0 citations

Jun 2026

Mural: Transferring LLM knowledge to image generation via Mixture-of-Transformers

Leveraging capabilities of large language models (LLMs) in text-to-image (T2I) synthesis is an important research direction. In this work we investigate whether the knowledge of a frozen LLM can be effectively utilized in T2I generation when trained exclusively on standard text-image pairs. We integrate a frozen, reaso...

Achin Jain, Jie An, Siddharth Chaudhary et al. · arXiv.org · 0 citations

Preprint Jul 2026

USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning

Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision-language models (e.g., CLIP) on downstream tasks. Among existing CLIP-based TTA methods, Test-Time Prompt Tuning (TPT) is a pioneering work that optimizes textual prompts using multiple test-time augmentations and remai...

Siru Jiang, Jian Liang, Ran He et al. · 0 citations

Preprint Jul 2026

DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example, is proposed and effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.

Zhen Huang, Yikun Wang, Shijie Xia et al. · 0 citations

Jun 2026

Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist

Faithfulness -- how precisely a generated image aligns with its prompt -- is increasingly central to the real-world utility of text-to-image (T2I) models. Existing faithfulness benchmarks, however, rely on simple atomic instructions, on which top-tier systems already achieve near-perfect scores. As T2I models enter cre...

Yuanhao Ban, Tong Xie, Sohyun An et al. · arXiv.org · 0 citations