Enabling Parallelism Hot Switching for Efficient Training of Large Language Models

作者：Houyang Ge, Fangcheng Fu, Haoyang Li, Xuanyu Wang, Sheng Lin, Y X Wang, Xiaonan Nie, Hailin Zhang, Xupeng Miao, Bin Cui · 年份：2024 · DOI：10.1145/3694715.3695969 · 被引用次数：5 · 研究领域：Topic Modeling、Advanced Neural Network Applications、Speech Recognition and Synthesis

Training of large-scale deep learning models necessitates parallelizing the model and data across numerous devices, and the choice of parallelism strategy substantially depends on the training workloads such as memory consumption, computation cost, and communication cost. Current approaches generally assume uniform training workloads across samples in a given task. Thus, existing systems are designed to adopt a static parallelism strategy throughout one training process. Nevertheless, when training models with sequence inputs, this assumption fails due to the sequence length variation across samples. Consequently, training with a static parallelism strategy would result in sub-optimal performance.