Scalable and Robust Reinforcement Learning Through Expectile Regression.

Abstract

Robust reinforcement learning (RRL) aims to develop a robust policy that maintains stable performance across diverse environments characterized by an uncertainty set. This set consists of perturbed environments derived from a nominal (training) environment that generates samples, thereby capturing potential discrepancies between training and real-world conditions. Recently, an adjacent uncertainty set has been introduced, providing more realistic perturbations compared to conventional formulations. Despite its solid theoretical foundation, the existing sample-based implementation of the robust Bellman update suffers from limited scalability and practical applicability in real-world scenarios. In this brief, we present, for the first time, scalable RRL algorithms that overcome these challenges by leveraging expectile regression. Extensive experiments demonstrate that the proposed methods significantly enhance the robustness of state-of-the-art (SOTA) RL algorithms while maintaining a practical computational cost comparable to strong off-policy baselines. In particular, the proposed methods exhibit up to a 23.6% average improvement in robustness under environmental perturbations over SOTA RL baselines while maintaining comparable computational complexity.