Kimi k1.5: Scaling Reinforcement Learning with LLMs

作者：Kimi Team, Angang Du, Bofei Gao, Bowei Xing, C. H. Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Liao, Chonghua, Tang, Chuning, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Wei, Guangda, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao-Tsung Yang, Hao Zhang, Haotian Yao, Zhao, Haotian, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, J. Zhao, Jin Zhang, Liu, Jingyuan, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Dong, Mengnan, Zhang, Neo, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Cao, Sihan, Siying Huang, Tao Jiang, Weihao Gao, Xiong, Weimin, W. He, Weixiao Huang, Xu, Weixin, Wenhao Wu, Wenyang He, Wei, Xianghui, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, X. H. Zhou, Xuehai Pan, Y F Young Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Y. Liu, Qin, Yidao, Yifeng Liu, Yingguo Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Z B Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Wang, Zhexu, Zhilin Yang, Zhiqi Huang, Huang, Zihao, Zhao Xu, Zonghan Yang, Lin, Zongyu · 发表于：arXiv (Cornell University) · 年份：2025 · DOI：10.48550/arxiv.2501.12599 · 被引用次数：11 · 研究领域：Robot Manipulation and Learning、Machine Learning and Data Classification、Machine Learning and Algorithms

Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement learning (RL) unlocks a new axis for the continued improvement of artificial intelligence, with the promise that large language models (LLMs) can scale their training data by learning to explore with rewards. However, prior published work has not produced competitive results. In light of this, we report on the training practice of Kimi k1.5, our latest multi-modal LLM trained with RL, including its RL training techniques, multi-modal data recipes, and infrastructure optimization. Long context scaling and improved policy optimization methods are key ingredients of our approach, which establishes a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models. Notably, our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities -- e.g., 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, 74.9 on MathVista -- matching OpenAI's o1. Moreover, we present effective long2short methods that use long-CoT techniques to improve short-CoT models, yielding state-of-the-art short-CoT reasoning results -- e.g., 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench -- outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%).