Skip to content

Frontier Learning: Training LLM Reasoners at the Edge of Capability

Sep 2026 · arXiv.org · Vol abs/2609.35426 · 0 citations · 48 references
Computer Science

TL;DR

Across several reasoning tasks and model families, the proposed frontier learning approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.

Abstract

Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.

View source

Similar papers

#natural language process... Open access Sep 2026

Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) has shown remarkable success in improving the mathematical reasoning of large language models. Yet problems beyond the model's current capability, where rollouts uniformly fail and no learning signal is produced, are structurally wasted despite marking the most info...

Yu-Kang Zhu, Zhen-Mao Han · 0 citations
#artificial intelligence Preprint Oct 2026

Sharpening Tax in Post-Training

An emerging hypothesis about reinforcement learning (RL) post-training of large language models (LLMs) is that it merely sharpens existing behaviors of a base model, improving single-shot accuracy at the cost of solution coverage. Although this trade-off has been observed in math and coding tasks, it need not extend to...

Changdae Oh, Qi Zeng, Qi Qi et al. · 1 citation
#machine learning Preprint Aug 2026

Demystifying Reinforcement Learning Post-Training of Language Models

This work deconstructs the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface, and uses the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shape...

D. Clay, Saket Gollapudi, Sankar V Harilal et al. · 1 citation
Preprint Aug 2026

Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies

Evolution Strategies (ES), a population-based, gradient-free post-training method that optimizes directly in weight space through random perturbations, achieves consistently higher pass@k than RL and produces a broader output distribution with greater solution coverage.

Conor F. Hayes, Elliot Meyerson, Kajetan Schweighofer et al. · 2 citations
#machine learning Preprint Aug 2026

PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs

PAC, a Progress-Augmented Advantage Curriculum for multi-task RL of LLMs that combines two task-level signals: advantage-derived learnability, which measures the magnitude of the policy update a task can induce, and recent reward gains, which show whether those updates have improved task performance.

Yuan-Qiang Yu, Yan-Zhao Zheng, Zhen-Tao Zhang et al. · 1 citation
Preprint Aug 2026

TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

A simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution, and introduces a lightweight diagnostic for identifying settings where TailSFT is most likely to help.

Sadhika Malladi, Samy Jelassi, Dylan J. Foster et al. · 3 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.