Multitask discriminator Proximity-Guided IRL (MPG) is introduced, which learns two complementary reward components: a generalizable discriminator that transfers shared structure across related tasks to identify expert behavior in a new task and a proximity function that measures how far a state deviates from expert behavior and provides corrective guidance during exploration.
Abstract
Inverse reinforcement learning (IRL) provides a powerful framework for learning from demonstrations. However, real-world tasks often exhibit substantial natural variations (e.g., picking up mugs with varying shapes), making it impractical to collect demonstrations that fully specify a new task under every possible scenario. In practice, while demonstrations for the target task are limited, it is often easier to obtain datasets of heterogeneous but related behaviors. This motivates the problem of few-shot IRL with multi-task demonstrations (FM-IRL), where an agent must learn a new task with substantial variations from only a limited number of target-task demonstrations, together with sufficient demonstrations of related tasks and online agent experience. To do so, we must both recover the expert distribution of the new task and provide guidance when the agent deviates from it. We introduce Multitask discriminator Proximity-Guided IRL (MPG), which learns two complementary reward components: (1) a generalizable discriminator that transfers shared structure across related tasks to identify expert behavior in a new task, and (2) a proximity function that measures how far a state deviates from expert behavior and provides corrective guidance during exploration. We demonstrate the effectiveness of our method on multiple challenging navigation and manipulation tasks under significant variations (e.g., object configurations, table layouts, and initial robot poses), achieving an average success rate of 81.2%, outperforming the strongest per-task baseline by an average of 24.7 percentage points.
A major bottleneck in imitation learning (IL) is the requirement of a large number of expert demonstrations, which can be expensive or even impossible to collect. Learning from a few expert demonstrations in addition to many supplementary potentially suboptimal demonstrations has emerged as a powerful paradigm to address this challenge. However, previous methods often fail to fully utilize the potential of these additional demonstrations by simply discarding nonexpert data. In contrast, our key insight is that suboptimal demonstrations can still be leveraged for policy training if they outperform the current policy. To realize this idea, we propose a novel approach named IL via meta-learning an action ranker (ILMAR), which performs weighted behavior cloning (BC) on a limited set of expert demonstrations along with supplementary demonstrations. It utilizes the functional of the advantage function to selectively integrate knowledge from the supplementary demonstrations. To make more effective use of supplementary demonstrations, we introduce a meta-goal in ILMAR to optimize the functional of the advantage function by explicitly minimizing the Kullback-Leibler (KL) divergence between the current policy and the expert demonstrations. Comprehensive experiments across a wide range of tasks demonstrate that ILMAR significantly outperforms previous methods in handling suboptimal demonstrations. The code is available at https://github.com/F-GOD6/ILMAR.
Jiang-Dong Fan, Guangyi Zhao, Hongcai He et al.· IEEE Transactions on Neural...· 0 citations
It is shown that the recovered reward improves a base policy without a supervised loss and yields further gains when optimized after standard supervised fine-tuning and can be used for contextual alignment, in which a single policy can be tailored to the preferences of different audiences.
Michal Wilinski, Le-Qi Liu, Chirag Nagpal· arXiv.org· 0 citations
This work develops LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies.
Manith Adikari, Bei Peng, Samuele Vinanzi et al.· arXiv.org· 0 citations
RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most, and substantially outperforms naive GRPO+OPD.
Zhuo-Wen Han, Jinwei Xiao, Zhengxi Lu et al.· 4 citations
River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization is proposed, which achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks.
Yi-Fan Yao, Bo Pang, Xuan-Phi Nguyen et al.· 1 citation
Action-conditioned world models are a key component of embodied AI, serving as scalable policy evaluators that reduce reliance on expensive real-world rollouts. To accurately capture diverse action-induced dynamics, such models should satisfy three key objectives-Physical Plausibility (P), Action Adherence (A), and Visual Fidelity (V), collectively referred to as PAV-while remaining robust to both in-distribution (ID) expert demonstrations and out-of-distribution (OOD) actions. However, existing methods primarily rely on ID action-video pairs and pixel-level reconstruction losses, which do not explicitly optimize PAV objectives and generalize poorly beyond expert data. To address this, we propose PAVXploreRL, a reinforcement learning framework built on a pretrained latent world model that explicitly optimizes PAV objectives through reward-driven training. To improve action generalization, our method jointly leverages ID trajectories and noise-driven OOD action exploration, without paired video supervision. Experiments show that PAVXploreRL consistently outperforms pretrained baselines, achieving a 5.6% average gain across benchmarks and producing higher-quality PAV properties. As a policy evaluator, it also yields more reliable performance estimates and reduces the overestimation bias of prior expert-only world models such as Ctrl-World. Code: https://github.com/Social-AI-Studio/PAVXploreRL
