A perturbed utility Markovian equilibrium (PUME) framework that preserves the scalability of link-based Markovian traffic equilibrium models and extends their applicability to settings with boundary choice probabilities, undiscounted network loading, and general link interactions.
Abstract
Large-scale traffic assignment requires equilibrium models that are both behaviorally plausible and computationally tractable. This paper develops a perturbed utility Markovian equilibrium (PUME) framework that preserves the scalability of link-based Markovian traffic equilibrium models and extends their applicability to settings with boundary choice probabilities, undiscounted network loading, and general link interactions. As the behavioral basis of PUME, we first develop the perturbed utility Markovian choice model (PUMCM) in which the Bellman optimality operator is defined through a convex surplus function whose gradient directly yields the optimal policy. The model generalizes existing additive random utility (ARUM) Markovian choice models and admits both interior and boundary choice probabilities. Accordingly, unattractive links can receive zero flow without imposing ex ante choice-set restrictions as in existing ARUM models. We establish conditions under which the corresponding Markov decision problem is well posed and yields a proper demand mapping. We then formulate the equilibrium as a variational inequality (VI) problem on the dual cost space and establish its existence and uniqueness. Particularly, the VI formulation of PUME accommodates non-separable and asymmetric cost structures and thus offers a more flexible modeling framework than existing Markovian traffic equilibrium (MTE) models. For computation, we develop a modified policy iteration method for network loading and a safeguarded accelerated meta-algorithm for computing equilibrium. Both algorithms are proven to be globally convergent and have demonstrated satisfactory numerical performances. Experiments on benchmark and synthetic networks further show that the proposed framework is highly scalable and robust towards a wide variety of demand-supply settings.
Standard solution concepts for stochastic games, such as Markov perfect equilibrium and Markov coarse correlated equilibrium, are computationally difficult, and thus, standard decentralized reinforcement-learning algorithms should not generally be expected to converge to them. In this paper, we study the equilibrium generated by such algorithms. In particular, we introduce a new solution concept for stochastic games, Markov Bayes coarse correlated equilibrium (MBCCE), defined as a distribution over states and stationary policy profiles such that, after observing the state but before observing her recommended action, no player can gain by choosing a different current action, with the sampled policy profile governing play thereafter. We discuss the parallels between MBCCE and coarse correlated equilibrium (CCE) in finite normal-form games and show that MBCCE retains several of its key properties. We then introduce a corresponding regret notion, adaptive Markov coarse regret (AMCR), and show that vanishing AMCR implies that every accumulation point of the empirical distribution of realized states and policy profiles is an MBCCE. Crucially, we show that achieving AMCR reduces to two standard learning tasks: minimizing external regret at each state and accurately evaluating the current joint policy. We then prove that under mild conditions these properties hold for two natural RL algorithmic designs: a decentralized asynchronous actor--critic algorithm through a new two-timescale stochastic-approximation analysis, and a standard episodic multi-agent projected policy-gradient method. Hence, both algorithms generate approximate MBCCEs, and we establish explicit finite-time convergence rates for both.
A duality-based characterization of implementability of dynamic edge flows for the multi-source, multi-destination case and a non-trivial proof that this assumption is always fulfilled for finitely supported edge flows with costs representing weighted travel times are provided.
We study infinite-horizon time-inconsistent Markov decision processes with a countably infinite state space and unbounded reward functions. The reward is allowed to depend explicitly on the initial time and initial state, thereby accommodating general sources of time inconsistency. We seek relaxed feedback equilibria, and our approach is based on entropy regularization and weighted functional analytic methods. With entropy regularization, we characterize a regular relaxed equilibrium through a fixed-point operator. By introducing two weight functions with distinct roles, one controlling the growth of rewards and values and the other defining the ambient weighted space, we construct a compact invariant set under a product topology and apply the Schauder-Tychonoff fixed-point theorem to establish existence of regularized equilibria. Importantly, the invariant set can be chosen uniformly for small entropy weight $\lambda\in(0,1]$. We then let $\lambda\to0+$ and show, through compactness, concentration of Gibbs policies, and uniform-integrability arguments, that a subsequential limit is a relaxed equilibrium of the original unregularized problem. We further study a policy iteration algorithm (PIA) for the entropy-regularized equilibrium problem. Under a weighted-discounting structure and sufficiently strong discounting, we establish exponential convergence and uniqueness of the regularized equilibrium in a suitable weighted Banach space. Combining the policy-iteration error with a quantitative soft-max approximation bound, we show that the iterated policies constitute weighted $\varepsilon$-equilibria for the original unregularized problem and derive an explicit regret estimate. A numerical example illustrating the convergence of PIA under strong discounting and a counterexample demonstrating its failure under weak discounting are also provided.
Natural Policy Gradient (NPG) is a well-established Reinforcement Learning algorithm that underlies widely used methods such as Trust Region Policy Optimization and Proximal Policy Optimization, both of which have demonstrated strong empirical success. In this paper, we study exact NPG in finite-horizon Markov Decision Processes with known dynamics and horizon-dependent transition kernels. We provide the first finite-time convergence guarantees for this algorithm in this setting, for which we consider both constant and increasing step size regimes. With a constant step size $\eta_t=\eta$, we prove that NPG converges sublinearly with a rate of $\mathcal{O}(H^{2}/t)$ after $t$ iterations, where $H$ is the horizon length. We also extend this constant step size analysis to linear MDPs in an exact population-projection oracle under a full support projection distribution, recovering the same sublinear rate as in the tabular setting. Furthermore, with increasing step sizes, we prove that this algorithm achieves a linear convergence rate of $\mathcal{O}\left(\left(1-\frac{1}{\vartheta_\rho}\right)^t\right)$ for a problem-dependent constant $\vartheta_\rho>1$, and the horizon-only robust schedule of the form $\eta_t=\eta_0(H/(H-1))^t$ where $\eta_0>0$ and $H \geq 2$, attains this same geometric rate.
Asha Barua, S. Khodadadian· arXiv.org· 0 citations
We study an insurance contract-design problem under moral hazard, endogenous participation, and strategic risk interdependence. Because the resulting $N$-agent game suffers from the curse of dimensionality, we approximate the strategic interactions via a heterogeneous mean-field game. We rigorously establish the existence of a lower-level mean-field Nash equilibrium using measurable selection arguments and the Kakutani fixed-point theorem. By proving the $L^1$-Lipschitz continuity of the aggregate participation threshold, we further establish equilibrium uniqueness via a contraction mapping. We then embed this mean-field response into the insurer's upper-level Stackelberg optimization problem. We formulate the objective through general performance envelopes to accommodate potential equilibrium multiplicity, proving the existence of upper-level $\varepsilon$-optimal contracts, and demonstrating the existence of an exact Stackelberg equilibrium under the uniqueness regime. We conclude by extending the model to finite contract menus, providing numerical evidence that multi-contract screening improves the principal's expected payoff in interdependent risk environments.
We study a simulation-based equilibrium problem arising in competitive insurance markets under hurricane risk. Each insurer seeks to maximize its own profit by selecting regional pricing and reinsurance decisions while satisfying insolvency constraints. The resulting problem is particularly challenging because customer purchase decisions induce discontinuous demand functions, while insolvency constraints create nonconvex feasible regions. To address these challenges, we introduce a pricing-dependent reinsurance optimization operator that reoptimizes reinsurance for each candidate pricing vector, transforming each insurer's joint response problem into a structured pricing problem in which insolvency feasibility is handled through reinsurance reoptimization. This reformulation avoids simultaneous optimization of pricing and reinsurance over the nonconvex joint feasible set. We optimize the resulting nonsmooth and nonconvex reduced objective using a trust-region framework that exploits both the smoothed and original objectives while incorporating direct-search exploration. The computed approximate responses are embedded within a damped better-response scheme to stabilize the equilibrium iterations. In a case study of the North Carolina hurricane insurance market, the framework identifies multiple equilibrium candidates with practical runtimes, while radius-based local Nash tests find no profitable sampled deviations within the tested neighborhoods.
Yunsoo Ha, Linda Nozick· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
