Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent

作者：Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Han, Jonny, Shu, Xiaobo, Jiahao Bu, Z C Chen, X. P. Huang, Fengzong Lian, Yang, Saiyong, Jianfeng Yan, Yuyuan Zeng, Xiaoqin Ren, Chao Yu, Lulu Wu, Yue Mao, Jun Xia, Tao Yang, Suncong Zheng, Kai Wu, Dian Jiao, Jinbao Xue, Xipeng Zhang, Decheng Wu, Kai Liu, Dan Wu, Guanghui Xu, Shaohua Chen, Shuang Chen, Xiao Feng, Yigeng Hong, Junqiang Zheng, Chengcheng Xu, Zongwei Li, Kuang, Xiong, Jiaochan Hu, Yiqi Chen, Yuchi Deng, Guiyang Li, Ao Liu, Chenchen Zhang, Shihui Hu, Zilong Zhao, Zifan Wu, Yao Ding, Weichao Wang, H. B. Liu, Wang, Roberts, Fei Hao, Peijie Yu, Ze Zhao, Xun Cao, Hai Wang, Xiang, Fusheng, Mengyuan Huang, Zhiyuan Xiong, Bin Hu, X. T. Hou, Lei Jiang, Jianqiang Ma, Jiajia Wu, Yaping Deng, Yi Shen, Qian Wang, Weijie Liu, Jie Liu, Meng Chen, Liang Dong, Jia, Weiwen, Chen Hu, Feifei Liu, Rui Yuan, Huilin Xu, Z.-Y. Yan, Tengfei Cao, Zhichao Hu, Xin‐Hua Feng, Du Dong, Tao Yu, Yangyu Tao, Feng Zhang, Jianchen Zhu, Chengzhong Xu, Xirui Li, Zha, Chong, Wen Ouyang, Yinben Xia, Xiang Li, Zhidong He, Chen, Rongpeng, Jiawei Song, Ruibin Chen, Fan Jiang, Chongqing Zhao, Wei Wang, Hao Gong, Rong Z. Gan, Weisheng Hu, Zhanhui Kang, Yong Yang, Yuhong Liu, Di Wang, Jie Jiang · 发表于：arXiv (Cornell University) · 年份：2024 · DOI：10.48550/arxiv.2411.02265 · 被引用次数：7 · 研究领域：Age of Information Optimization

In this paper, we introduce Hunyuan-Large, which is currently the largest open-source Transformer-based mixture of experts model, with a total of 389 billion parameters and 52 billion activation parameters, capable of handling up to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior performance across various benchmarks including language understanding and generation, logical reasoning, mathematical problem-solving, coding, long-context, and aggregated tasks, where it outperforms LLama3.1-70B and exhibits comparable performance when compared to the significantly larger LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale synthetic data that is orders larger than in previous literature, a mixed expert routing strategy, a key-value cache compression technique, and an expert-specific learning rate strategy. Additionally, we also investigate the scaling laws and learning rate schedule of mixture of experts models, providing valuable insights and guidances for future model development and optimization. The code and checkpoints of Hunyuan-Large are released to facilitate future innovations and applications. Codes: https://github.com/Tencent/Hunyuan-Large Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large