Integrating AIRL-Based Reward Estimation into a Control Method Using Fuzzy Inference and Policy Gradient

Abstract

A method integrating policy gradient with fuzzy inference has been proposed. This method, hereafter referred to as the base model, has high flexibility with respect to the configuration of fuzzy inference rules and fuzzy representation. It can also automatically adjust the rule weights and the membership functions in the antecedent and consequent parts using the same framework for learning. In methods that integrate reinforcement learning with fuzzy inference, however, the reward function must be designed in advance. It is therefore useful to add learning-based reward estimation to the base model. This paper proposes a fuzzy control method that integrates a reward estimation mechanism based on the adversarial inverse reinforcement learning (AIRL) framework into the base model. The generative adversarial nets (GAN) framework adopted in AIRL, however, tends to suffer from unstable learning. To address this, the proposed method treats the policy of the base model as the generator and also uses the set of membership functions in the policy as the cost function of the discriminator. This configuration allows the discriminator update for reward estimation and the generator update for policy estimation to be performed simultaneously on the parameters shared by both, without estimating a value function. In simulations of an illustrative vehicle speed control task, the proposed method acquired policies with high generalization performance through stable and efficient learning.