Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding

作者：StepFun Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Hao Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, Song Yuan, Wuxun Xie, Xiaoniu Song, Xing Chen, Xingping Yang, Xuelin Zhang, Yanbo Yu, Yaoyu Wang, Yibo Zhu, Yimin Jiang, Yu Zhou, Yuanwei Lu, Houyi Li, Jingcheng Hu, Kalam Lo, Ailin Huang, Binxing Jiao, Bo Li, Boyu Chen, Changxing Miao, Chang Lou, Chen Hu, Chen Xu, Chen-Xi Yu, Chengyuan Yao, Da-Wei Lv, Da Shi, Deshan Sun, Ding Huang, Dingyuan Hu, Dongqing Pang, Enle Liu, Fajie Zhang, Fanqi Wan, Gulin Yan, Hanshan Zhang, Han Zhou, Hang Wu, Hangyu Guo, Hanqiu Chen, Hanshan Zhang, Hao Wu, Haochen Zhang, Haolong Yan, Haoran Lv, Haoran Wei, Hebin Zhou, Heng Wang, Hongxin Li, Hongyu Zhou, Hongyuan Wang, Huiyong Guo, Jia Wang, Jiahao Gong, Jia-Jie Xie, Jian Zhou, Jian‐Yuan Sun, J. Wu, Jiaran Zhang, Jiayu Liu, Jie Cheng, Jie Luo, Jie Yan, Jie Yang, Jieyi Hou, Jinguang Zhang, Jinlan Cao, Jisheng Yin, Junfeng Liu, Junhao Huang, Junzhe Lin, Kaijun Tan, Kai-yang Li, Kang An, Kangheng Lin, Ke-Chun Liu, Lei Yang, Liang Zhao, Liangyu Chen, Lieyu Shi, L. Tan, Lin Lin, Lin Zhang, Lina Chen, Liwen Huang, Li-Li Shi, Longlong Gu, Mei Chen, Meng-Yao Ren, Ming Li, Mingzhe Chen, Na Wang, Nan Wu, Qi Han, Qian Zhao, Qiang Zhang, Qian Liu, Qiao-Li Chen, Qiling Wu, Qi He, Qi-Liang Tan, Qiufeng Wang, Qiuping Wu, Qiuyan Liang, Quan Sun, Rui Li, Rui Miao, Ruosi Wan, Ruyan Guo, Shangwu Zhong, Shaoliang Pang, Sheng Fan, Shijie Shang, Shi‐Yong Jiang, S. Yang, Shiming Hao, Shu-Guang Gao, S. Huang, Siqi Liu, Tiancheng Cao, Tianhao Cheng, Tian-Fang Peng, Wang You, Wei Ji, Wencheng Sun, Wen-Yu Deng, Wenqing He, Wen-gang Zheng, Xi Chen, Xiang-Zhen Kong, Xianzhen Luo, Xiaobo Yang, Xiaojian Liu, Xiao‐Dong Ren, Xin-hao Han, Xin Li, Xin Wu, Xuefeng Zhao, Yanan Wei, Yang Li, Yangguang Li, Yang Xu, Yanming Xu, Ya-jun Shi, Yeqing Shen, Yi Yang, Yifei Yang, Yifeng Gong, Yihan Chen, Yijing Yang, Yinmin Zhang, Yizhuang Zhou, Yuanhao Ding, Yuantao Fan, Yuanzhen Yang, Yu Luo, Yue Peng, Yufan Lu, Yu-Ying Deng, Yuhe Yin, Yujie Liu, Yukun Chen, Yuling Zhao, Yunfei Mou, Yunlong Li, Yunzhou Ju, Yusheng Li, Yuxiang Yang, Yuxiang Zhang, Yuyang Chen, Zejia Weng, Zhengyan Xie, Zheng Ge, Zheng Gong, Zhen-Rong Lu, Zhewei Huang, Zhichao Chang, Zhiguo Huang, Zhirui Wang, Zi-yi Yang, Zili Wang, Ziqi Wang, Zixin Zhang, Daxin Jiang, H. Shum, Xiangyu Zhang · 发表于：arXiv.org · 年份：2025 · DOI：10.48550/arXiv.2507.19427 · 被引用次数：56 · 研究领域：Computer Science

Large language models (LLMs) face low hardware efficiency during decoding, especially for long-context reasoning tasks. This paper introduces Step-3, a 321B-parameter VLM with hardware-aware model-system co-design optimized for minimizing decoding costs. Step-3 innovates in two key dimensions: (1) A novel Multi-Matrix Factorization Attention (MFA) mechanism that significantly reduces both KV cache size and computation while maintaining high attention expressiveness, and (2) Attention-FFN Disaggregation (AFD), a distributed inference system that decouples attention and Feed-Forward Network (FFN) layers into specialized subsystems. This co-design achieves unprecedented cost efficiency: Step-3 significantly reduces theoretical decoding costs compared with models like DeepSeek-V3 and Qwen3 MoE 235B, with the gains widening at longer context. Step-3 achieves low cost while activating 38B parameters per token (more than DeepSeek-V3 and Qwen3 MoE 235B), demonstrating that hardware-aligned attention arithmetic intensity, MoE sparsity, and AFD are critical to cost-effectiveness. We perform a head-to-head comparison with DeepSeek-V3 in its favorable scenarios. Our implementation on Hopper GPUs achieves a decoding throughput of up to 4,039 tokens per second per GPU under 50ms TPOT SLA (4K context, FP8, no MTP). It is higher than DeepSeek-V3's 2,324 in the same setup and sets a new Pareto frontier for LLM decoding.