Skip to content

UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents

Sep 2026 · 3 citations · ⚡ 1 influential · 50 references
Computer Science

TL;DR

Empirically, UOPD is evaluated across a broad range of agentic tasks, including ALFWorld, WebShop, and Search, demonstrating its superior performance over OPD methods and their variants.

Abstract

On-policy distillation (OPD) trains a student on its own rollouts using dense supervision from a teacher. In multi-turn environments, a mistake at a critical decision step can redirect the subsequent rollout toward poor outcomes. We use low teacher confidence on student actions to select high-uncertainty steps for correction. In a controlled ALFWorld study, a single teacher correction at a low-confidence step improves subsequent student behavior and task success, motivating selective intervention during distillation. We propose UOPD, an uncertainty-aware intervention method for on-policy distillation. At low-uncertainty turns, UOPD executes student actions and applies the standard OPD loss. At high-uncertainty turns, it samples and executes teacher actions and trains the student to imitate them through supervised fine-tuning, which minimizes forward Kullback-Leibler divergence in expectation. UOPD utilizes adaptive uncertainty thresholds to target a scheduled intervention rate. Empirically, we evaluate UOPD across a broad range of agentic tasks, including ALFWorld, WebShop, and Search, demonstrating its superior performance over OPD methods and their variants. UOPD improves WebShop score by up to $15.8\%$ relative to standard OPD.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Stochastic Teacher Intervention for Agentic On-Policy Distillation

STI-OPD, a stochastic teacher intervention framework for multi-turn agentic OPD, and an Importance-Weighted Reverse KL objective that corrects the token sampling mismatch between teacher-generated responses and the student policy to preserve the original OPD objective are introduced.

Jun-Nan Liu, Lin-Hao Luo, Zhijun Chen et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation

This work identifies Teacher Uncertainty Contraction (TUC), a systematic phenomenon whereby the teacher's predictive uncertainty decreases as it continues from a student-generated prefix, and proposes Adaptive-Continuations On-Policy Distillation (AC-OPD), which augments informative states along student rollouts with t...

Jin-Gang Zhou, Yu-Yi Zhou, Hai-Yang Guo et al. · 3 citations
#artificial intelligence Preprint Oct 2026

DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

On-policy distillation (OPD) provides dense token-level supervision on student rollouts, but this supervision can become unreliable when the student and teacher diverge, either because of a large initial policy gap or compounding errors in long-horizon reasoning and tool use. We introduce Divergence-Triggered Dynamic D...

Qi-Jia He, Ze-Qun Li, S.-S. Luo et al. · 0 citations

Reward-Aligned Reweighting for On-Policy Distillation

Reward-Aligned Reweighting for On-Policy Distillation (R$^{2}$-OPD), which uses outcome agreement and the magnitude of teacher--student disagreement to continuously reallocate teacher supervision, achieves the highest average accuracy among the compared training methods in both cross-size and same-size distillation.

Hao Xu, Junwei Su, Lan-Song Diao et al. · 0 citations
#machine learning Preprint Oct 2026

Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation

On-policy self-distillation (OPSD) supplies dense token-level feedback on trajectories sampled from the student's own policy, a richer training signal than the outcome-level rewards of reinforcement learning. This feedback comes from a teacher conditioned on a full reference solution unavailable to the student. The ref...

Rui Li, Li-Yang He, Zheng Zhang et al. · 0 citations
#machine learning Preprint Oct 2026

State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents

On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models, while extending its success to multi-turn agentic environments remains challenging. We study agentic OPD through interaction states, distinguishing the objective task and execu...

Zhen-Tao Zhang, Yan-Zhao Zheng, Yan-Lin Liu et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.