AdanB: Adan with Dynamic Bound for Deep Learning

作者：Zhiyang Zhou, Huisheng Zhang · 年份：2024 · DOI：10.1109/ntci64025.2024.10776348 · 被引用次数：2 · 研究领域：Anomaly Detection Techniques and Applications、Computational Physics and Python Applications

The adaptive Nesterov Momentum Algorithm (Adan) is a recently proposed efficient adaptive optimization algorithm that innovatively combines the Adam algorithm with an improved version of the NAG method to produce superior performance. However, Adan still suffers from unstable learning rates near convergence due to the second-order moments. To this end, this paper proposes a modified Adan with dynamic bound (AdanB) by employing a clipping trick. AdanB achieves a smooth transition from Adan to SGD by introducing dynamic learning rate boundaries, thereby improving the model's generalization ability while maintaining high learning speed. Extensive simulations are conducted to confirm the superiority of AdanB over Adan when training three popular neural models on two benchmark image classification problems.