Off-Policy Inverse Q-Learning Algorithm with Unobservable States
TL;DR
This paper investigates the output-feedback inverse reinforcement learning (IRL) problem and proposes a data-driven output-feedback off-policy inverse Q-learning algorithm that relies solely on input–output trajectory data from an expert system to address the problem of an unknown performance index in discrete-time linear systems with unobservable states.
Abstract
To address the problem of an unknown performance index in discrete-time (DT) linear systems with unobservable states, this paper investigates the output-feedback inverse reinforcement learning (IRL) problem and proposes a data-driven output-feedback off-policy inverse Q-learning algorithm. The proposed method relies solely on input–output trajectory data from an expert system. It does not require a system-dynamics model or state information to identify the unknown cost function and learn an optimal control policy. First, for cases where the system dynamics and the target control gain are known, we propose a model-based method. Building on this foundation, we further develop a model-free method that does not require a system-dynamics model or state information. The proposed algorithm uses state reconstruction to transform the state-feedback Q-function equation into an input–output form. Simulations using an F-16 aircraft model demonstrate that the learner system can approximate the expert trajectories and effectively reconstruct the performance-index function.
