MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training

作者：Lu Zhao, Rong Shi, Shaoqing Zhang, Yueqiang Chen, Bo He, Hongfeng Sun, Z. Yin, Shangchao Su, Zhiyan Cui, Liang Dong, X. Q. Li, Lingbin Wang, Jianwei He, J. Ma, Weikang Huang, Jincheng Tong, Dongdong Gao, Jianguo Zhang, Hong Tian, Hui Shen, Zongtai Luo, Zhaoqun Sun, Hongxing Niu, Yue Sun · 发表于：arXiv (Cornell University) · 年份：2025 · DOI：10.48550/arxiv.2511.21431 · 研究领域：Parallel Computing and Optimization Techniques、Advanced Neural Network Applications、Distributed and Parallel Computing Systems

The training of large-scale Mixture of Experts (MoE) models faces a critical memory bottleneck due to severe load imbalance caused by dynamic token routing. This imbalance leads to memory overflow on GPUs with limited capacity, constraining model scalability. Existing load balancing methods, which cap expert capacity, compromise model accuracy and fail on memory-constrained hardware. To address this, we propose MemFine, a memory-aware fine-grained scheduling framework for MoE training. MemFine decomposes the token distribution and expert computation into manageable chunks and employs a chunked recomputation strategy, dynamically optimized through a theoretical memory model to balance memory efficiency and throughput. Experiments demonstrate that MemFine reduces activation memory by 48.03% and improves throughput by 4.42% compared to full recomputation-based baselines, enabling stable large-scale MoE training on memory-limited GPUs.