Control-theoretic methods for reinforcement learning

Abstract

Reinforcement Learning (RL) has become a cornerstone of modern artificial intelligence and control theory, enabling complex decision-making tasks in a wide variety of domains such as robotics, autonomous driving and strategic games. While deep neural networks allow RL methods to address large-scale problems, exploiting the mathematical structures underlying these problems remains essential for designing efficient and theoretically grounded algorithms. This thesis bridges two complementary perspectives, mathematical analysis of reinforcement learning problems and control-theoretic principles, to develop a rigorous foundation for the design and the analysis of novel reinforcement learning algorithms. The first part of this work investigates a class of learning algorithms that generalize the classical RL paradigm of concurrent policy evaluation and policy improvement. Extending the classical two-timescale framework, a third concurrent process is introduced to incorporate system measurements. Using tools based on singular perturbations theory and LaSalle's invariance principle, sufficient conditions for the convergence of this class of methods are derived. Two novel actor-critic variants are then proposed to address two scenarios with different available measurements from the system. The second part focuses on reward-balancing methods, a recently proposed family of algorithms that approach RL from a dual viewpoint. Unlike standard methods that iteratively improve the policy while keeping the reward fixed, these methods fix the policy to be greedy and iteratively adjust the reward function to make that policy optimal, without altering the underlying problem. This thesis provides a detailed characterization of the transformation group underlying this process and introduces a control-theoretic reformulation of the approach. The resulting framework enables principled inclusion of constraints and penalty terms to improve robustness under model uncertainty. Overall, by exploiting the inherent generality of systems theory, this thesis takes a step toward a unifying framework for the design and the analysis of RL algorithms that leverage their underlying structural properties.