Realistic Actor-Critic: A framework for balance between value overestimation and underestimation

作者：Sicen Li, Qinyun Tang, Yiming Pang, Xinmeng Ma, Gang Wang · 发表于：Frontiers in Neurorobotics · 年份：2023 · DOI：10.3389/fnbot.2022.1081242 · 被引用次数：7 · 研究领域：Reinforcement Learning in Robotics、Explainable Artificial Intelligence (XAI)、Adversarial Robustness in Machine Learning

Introduction: The value approximation bias is known to lead to suboptimal policies or catastrophic overestimation bias accumulation that prevent the agent from making the right decisions between exploration and exploitation. Algorithms have been proposed to mitigate the above contradiction. However, we still lack an understanding of how the value bias impact performance and a method for efficient exploration while keeping stable updates. This study aims to clarify the effect of the value bias and improve the reinforcement learning algorithms to enhance sample efficiency. Methods: This study designs a simple episodic tabular MDP to research value underestimation and overestimation in actor-critic methods. This study proposes a unified framework called Realistic Actor-Critic (RAC), which employs Universal Value Function Approximators (UVFA) to simultaneously learn policies with different value confidence-bound with the same neural network, each with a different under overestimation trade-off. Results: This study highlights that agents could over-explore low-value states due to inflexible under-overestimation trade-off in the fixed hyperparameters setting, which is a particular form of the exploration-exploitation dilemma. And RAC performs directed exploration without over-exploration using the upper bounds while still avoiding overestimation using the lower bounds. Through carefully designed experiments, this study empirically verifies that RAC achieves 10x sample efficiency an...