OnActor-Critic Algorithms

作者：Vijay R. Konda, John N. Tsitsiklis · 发表于：SIAM Journal on Control and Optimization · 年份：2003 · DOI：10.1137/s0363012901385691 · 被引用次数：724 · 研究领域：Reinforcement Learning in Robotics、Markov Chains and Monte Carlo Methods、Advanced Control Systems Optimization

In this article, we propose and analyze a class of actor-critic algorithms. These are two-time-scale algorithms in which the critic uses temporal difference learning with a linearly parameterized approximation architecture, and the actor is updated in an approximate gradient direction, based on information provided by the critic. We show that the features for the critic should ideally span a subspace prescribed by the choice of parameterization of the actor. We study actor-critic algorithms for Markov decision processes with Polish state and action spaces. We state and prove two results regarding their convergence.