Compression Error Sensitivity Analysis for Different Experts in MoE Model Inference

作者：Songkai Ma, Zhaorui Zhang, Sheng Di, Benben Liu, Xiaodong Yu, Xiaoyi Lu, Dan Wang · 年份：2025 · DOI：10.1145/3731599.3767377 · 被引用次数：2 · 研究领域：Explainable Artificial Intelligence (XAI)、Bayesian Modeling and Causal Inference、Machine Learning and Algorithms

With the widespread application of Mixture of Experts (MoE) reasoning models in the field of LLM learning, efficiently serving MoE models under limited GPU memory constraints has emerged as a significant challenge. Offloading the non-activated experts to main memory has been identified as an efficient approach to address such a problem, while it brings the challenges of transferring the expert between the GPU memory and main memory. We need to explore an efficient approach to compress the expert and analyze how the compression error affects the inference performance.