Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination

作者：Ming Hu, Zhihao Yue, Xiaofei Xie, Cheng Chen, Yihao Huang, Xian Wei, Xiang Lian, Yang Liu, Mingsong Chen · 年份：2024 · DOI：10.1145/3637528.3671722 · 被引用次数：24 · 研究领域：Privacy-Preserving Technologies in Data、Cryptography and Data Security、Stochastic Gradient Optimization Techniques

Although Federated Learning (FL) enables global model training Xiaofei Xie xfxie@smu.edu.sg Singapore Management University Singapore, Singapore Xian Wei xwei@sei.ecnu.edu.cn East China Normal University Shanghai, China Mingsong Chen∗ mschen@sei.ecnu.edu.cn East China Normal University Shanghai, China • Computing methodologies → Distributed artificial intelligence. across clients without compromising their raw data, due to the unevenly distributed data among clients, existing Federated Averaging (FedAvg)-based methods suffer from the problem of low inference performance. Specifically, different data distributions among clients lead to various optimization directions of local models. Aggregating local models usually results in a low-generalized global model, which performs worse on most of the clients. To address the above issue, inspired by the observation from a geometric perspective that a well-generalized solution is located in a flat area rather than a sharp area, we propose a novel and heuristic FL paradigm named FedMR (Federated Model Recombination). The goal of FedMR is to guide the recombined models to be trained towards a flat area. Unlike conventional FedAvg-based methods, in FedMR, the cloud server recombines collected local models by shuffling each layer of them to generate multiple recombined models for local training on clients rather than an aggregated global model. Since the area of the f lat area is larger than the sharp area, when local models are located in...