Dynamic Programming for Reinforcement Learning with Large State Spaces – Hierarchical Abstraction

Abstract

Reinforcement learning (RL) has demonstrated remarkable success in various domains, yet its application is often hindered by the computational complexity associated with large state spaces. Traditional dynamic programming algorithms, such as Value Iteration and Policy Iteration, suffer severely from the curse of dimensionality, rendering them impractical for problems with a vast number of states. This paper proposes a novel approach to address this challenge by integrating hierarchical dynamic programming with learned abstraction layers. We decompose the state space into a hierarchy of abstraction levels, employing autoencoders to learn low-dimensional representations (embeddings) at each level. These embeddings facilitate efficient state aggregation and enable the application of dynamic programming within the hierarchical structure. The core idea is to reduce the problem size by representing the state space with a hierarchy of compact, learned representations, thereby mitigating the computational burden. This approach offers a scalable solution for RL problems with large state spaces, presenting a significant advancement in the field.