ByteScale: Communication-Efficient Scaling of LLM Training with a 2048K Context Length on 16384 GPUs

作者：Houyang Ge, Junda Feng, Qi Huang, Fangcheng Fu, Xiaonan Nie, Lei Zuo, Haibin Lin, Bin Cui, Xin Liu · 年份：2025 · DOI：10.1145/3718958.3754352 · 被引用次数：4 · 研究领域：Advanced Neural Network Applications、Parallel Computing and Optimization Techniques、Software Testing and Debugging Techniques

Scaling long-context ability is essential for Large Language Models (LLMs). To amortize the memory consumption across multiple devices in long-context training, inter-data partitioning (a.k.a. Data Parallelism) and intra-data partitioning (a.k.a. Context Parallelism) are commonly used. Current training frameworks predominantly treat the two techniques as orthogonal, and establish static communication groups to organize the devices as a static mesh (e.g., a 2D mesh). However, the sequences for LLM training typically vary in lengths, no matter for texts, multi-modalities or reinforcement learning. The mismatch between data heterogeneity and static mesh causes redundant communication and imbalanced computation, degrading the training efficiency.