Exercises in Mathematical Foundations for Reinforcement Learning: A Companion Workbook

Abstract

Reading a complete mathematical proof and constructing one are different skills. This workbook develops the second through structured practice with the mathematical foundations used in reinforcement learning. The volume contains 93 exercises aligned with Chapters 2–17 of Mathematical Foundations for Reinforcement Learning. It follows the mathematical companion’s chapter sequence, notation, and order of dependencies. Each exercise identifies the definitions and results needed to attempt it, allowing readers to move directly from studying the theory to applying it. The exercises ask readers to construct mathematical objects, verify assumptions, reconstruct proofs, diagnose and repair invalid arguments, and combine several tools in a single solution. Particular attention is given to the details that make mathematical arguments valid, including measurability, integrability, conditioning, convergence, and stability. Selected exercises connect these foundations to specific results in Reinforcement Learning: Theory and Algorithms, showing why the mathematics matters in reinforcement-learning analysis. Topics span measure and conditional probability, stochastic processes, matrix methods, martingales, stochastic approximation, concentration inequalities, information theory, Bayesian conditioning, convex analysis, and statistical and online learning. The workbook is intended for students, engineers, researchers, and independent readers seeking to progress from following mathematical arguments to constructing and checking them. It is designed for use alongside the mathematical companion, with the corresponding chapter studied before attempting its exercises. This deposit contains Version 1.0 of the completed exercises-only edition. Worked solutions are currently in preparation and are not included in this release. The planned solutions edition will retain the same exercise numbering and labels.