SignGD with error feedback meets lazily aggregated technique: Communication-efficient algorithms for distributed learning

作者：Xiaoge Deng, Tao Sun, Feng Liu, Dongsheng Li · 发表于：Tsinghua Science & Technology · 年份：2021 · DOI：10.26599/tst.2021.9010045 · 被引用次数：13 · 研究领域：Stochastic Gradient Optimization Techniques、Sparse and Compressive Sensing Techniques、Distributed Sensor Networks and Detection Algorithms

The proliferation of massive datasets has led to significant interests in distributed algorithms for solving large-scale machine learning problems. However, the communication overhead is a major bottleneck that hampers the scalability of distributed machine learning systems. In this paper, we design two communication-efficient algorithms for distributed learning tasks. The first one is named EF-SIGNGD, in which we use the 1-bit (sign-based) gradient quantization method to save the communication bits. Moreover, the error feedback technique, i.e., incorporating the error made by the compression operator into the next step, is employed for the convergence guarantee. The second algorithm is called LE-SIGNGD, in which we introduce a well-designed lazy gradient aggregation rule to EF-SIGNGD that can detect the gradients with small changes and reuse the outdated information. LE-SIGNGD saves communication costs both in transmitted bits and communication rounds. Furthermore, we show that LE-SIGNGD is convergent under some mild assumptions. The effectiveness of the two proposed algorithms is demonstrated through experiments on both real and synthetic data.