EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models

作者：Linglin Jing, Yuting Gao, Zhigang Wang, Lan Wang, Yiwen Tang, Wenhai Wang, Kaipeng Zhang, Qingpei Guo · 发表于：arXiv (Cornell University) · 年份：2025 · DOI：10.48550/arxiv.2505.23830 · 被引用次数：1 · 研究领域：Expert finding and Q&A systems、Topic Modeling、Computational and Text Analysis Methods

Recent advancements have shown that the Mixture of Experts (MoE) approach significantly enhances the capacity of large language models (LLMs) and improves performance on downstream tasks. Building on these promising results, multi-modal large language models (MLLMs) have increasingly adopted MoE techniques. However, existing multi-modal MoE tuning methods typically face two key challenges: expert uniformity and router rigidity. Expert uniformity occurs because MoE experts are often initialized by simply replicating the FFN parameters from LLMs, leading to homogenized expert functions and weakening the intended diversification of the MoE architecture. Meanwhile, router rigidity stems from the prevalent use of static linear routers for expert selection, which fail to distinguish between visual and textual tokens, resulting in similar expert distributions for image and text. To address these limitations, we propose EvoMoE, an innovative MoE tuning framework. EvoMoE introduces a meticulously designed expert initialization strategy that progressively evolves multiple robust experts from a single trainable expert, a process termed expert evolution that specifically targets severe expert homogenization. Furthermore, we introduce the Dynamic Token-aware Router (DTR), a novel routing mechanism that allocates input tokens to appropriate experts based on their modality and intrinsic token values. This dynamic routing is facilitated by hypernetworks, which dynamically generate routing we...