Discrete Action Offline Reinforcement Learning with Tangled Program Graphs

Abstract

Offline reinforcement learning (offline RL) has been shown to improve sample-efficiency and safety in learning control policies. So far, much of offline reinforcement learning research has been centered on Deep Reinforcement Learning (DRL), which results in black-box models. Given that offline RL is prone to failure when the agent encounters data outside of the state-action coverage of the offline dataset, there is a need for 'white box' verifiable solutions to offline RL to guarantee correctness of the policy. This work demonstrates that Tangled Program Graphs (TPG) can be used under an offline imitation learning regime to learn policies for 4 benchmark Gymnasium environments: CartPole, MountainCar, Acrobot and LunarLander from an offline dataset with no further interaction with the task environment after training. The results show that the policies learned offline are competitive with the black-box oracle policy. The symbolic policies are interpretable and parsimonious which suggests that black-box oracle policies may naturally distill from high-capacity networks into simple symbolic algorithms.