Discrete-Time Singularly Perturbed Markov Decision Processes with a General State Space

Abstract

Abstract. This paper studies singular perturbations of discrete-time Markov chains in general state spaces, which may include the transient set. By leveraging the idea of state aggregation, we derive a Taylor series expansion for the invariant probability measure of the singularly perturbed Markov chain. We then apply this expansion to analyze the expected long-run average cost (EAC) for singularly perturbed Markov decision processes (MDPs). For this EAC problem, we establish the existence of an optimal pure stationary policy and construct an explicit asymptotically optimal policy. A contribution of our work lies in the application of strong stability conditions (see [N. V. Kartashov, J. Soviet Math., 34 (1986), pp. 1493–1498]), allowing us to relax the widely used Doeblin conditions (see [S. P. Meyn and R. L. Tweedie, Markov Chains and Stochastic Stability, 2nd ed., Cambridge University Press, Cambridge, UK, 2009]) and thereby extend the theory’s applicability to a broader class of practical systems. The efficacy of our results is demonstrated through a maintenance system as an example.