Dual Advantage-Guided Offline Reinforcement Learning

Abstract

Offline reinforcement learning aims to learn effective policies from fixed datasets without online interaction, necessitating conservative constraints to mitigate the out-ofdistribution issue. Although existing approaches alleviate this issue through conservative constraints or policy regularization, they still struggle to effectively exploit high-quality samples in offline datasets. To address these issues, this study proposes Dual Advantage-Guided Offline Reinforcement Learning (DAG). The approach designs an Advantage-Guided Variational Autoencoder (AG-VAE) to reconstruct the behavior policy from offline datasets. The policy learns from both reconstructed actions and high-advantage dataset actions, with advantage weighting facilitating policy generalization. To evaluate the effectiveness of the proposed approach, DAG is assessed on MuJoCo robotic control tasks and AntMaze tasks from the D4RL benchmark across datasets with varying data quality. Experimental results demonstrate that DAG effectively exploits advantage information and outperforms existing offline reinforcement learning approaches across multiple tasks. Compared with the TD3+BC baseline, DAG improves the total normalized score by 9.69% on MuJoCo and 10.06% on AntMaze.