Probabilistic Programming for Reinforcement Learning with Exploration

Abstract

Reinforcement learning (RL) faces significant challenges in achieving optimal performance due to the inherent difficulty in balancing exploration and exploitation. This paper proposes a novel framework leveraging probabilistic programming (PP) to address this problem. The core idea is to represent an RL agent as a probabilistic program, enabling the explicit modeling of uncertainty within the environment and the agent itself. This allows for the incorporation of Bayesian approaches to exploration, where actions are actively sampled based on predicted rewards and a quantified measure of uncertainty. Unlike traditional RL methods that often rely on heuristics or point-estimate models, this approach provides a more principled and flexible framework for exploration, potentially leading to improved sample efficiency and overall performance, particularly in complex and partially observable environments. The framework utilizes concepts from Bayesian inference, Markov Decision Processes (MDPs), and probabilistic programming to define a coherent and powerful approach to RL. Key elements include the definition of probability distributions for state values, action values, and transition probabilities, and the application of variational inference or other inference techniques to estimate these distributions. This allows the agent to learn not just optimal actions, but also a representation of its knowledge about the environment.