FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging

作者：Zichen Tang, Ziyan Ma, Haoyang He, Jiacheng Liu, Zhongjun Yang, Zihua Rong, Rongjin Li, Kun Ji, Qing Huang, Xinyang Hu, Yang Liu, Qianhe Zheng, Mark Chen, Jerry Tworek, Hee-woo Jun, Qim-ing Yuan, Henrique Pondé, O. Pinto, Jared Ka-plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, A. Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas-try, Pamela Mishkin, Brooke Chan, Scott Gray, N. Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo Bavarian, Clemens Winter, P. Tillet, F. Such, Dave Cum-mings, Matthias Plappert, Fotios Chantzis, Eliza-beth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, A. Paino, N. Tezak, Jie Tang, Igor Babuschkin, S. Balaji, Shantanu Jain, William Saunders, Chris Hesse, Andrew N Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, M. Brundage, M. Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Wenhu Chen, Xueguang Ma, Xinyi Wang, William W. Cohen, Ming Yin, Max W.F. Ku, Pan Lu, Yixin Wan, Jianyu Xu, Tony Xia, TheoremQA, Zhiyu Chen, Charese H. Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, W. Wang, Shiyang Li, Zhiqiang Ma, William Yang, ConvFinQA, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, A. Liu, Bing Xue, Bing-Li Wang, Bochao Wu, B. Feng, Chengda Lu, Chenggang Zhao, C. Deng, Chenyu Zhang, C. Ruan, Damai Dai, Deli Chen, Dong-Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, JingChang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. Cai, J. Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, K. Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, M. Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Wen Liu, W. Liang, Wenjun Gao, Wen-xuan Yu, Wentao Zhang, W. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, X. Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Xiangyu Jin, Xi-Cheng Shen, Xiaosha Chen, Chengqi Zhao, Chenyu Deng, Chong Zhang, Ruan Damai, Daya Dai, Dejian Guo, Deli Yang, Dongjie Chen, Erhang Ji, Fangyun Li, Fucong Lin, Dai Fuli, Guangbo Luo, Guanting Hao, Guowei Chen, Han Zhang, Haocheng Xu, Haowei Wang, Honghui Zhang, Huajian Ding, Huazuo Xin, Hui Gao, J. Qu, Jiaqi Guo, Jiashi Ni, Jiawei Li, Wang Jin, Jingyang Chen, Jun-jie Yuan, Junlong Qiu, Junxiao Li, Kai Song, Kang Gao, Kexin Guan, Kuai Huang, Peiyi Huang, Peng Wang, Qiancheng Zhang, Wang Qihao, Qin-Feng Zhu, Qiushi Chen, R. Du, L. ChenR., Ruisong Ge, Zhang, Tian Pei, Tianyu Sun, Wangding Zeng, Wanjia Zhao, Xianzu Wang, Xiaokang Chen, X. Nie, Yunxian Xiong, Yuting Ma, Yuxiang Yan, Yuxi-ang Luo, Yuxuan You, Yuyang Liu, Z. Zhou, Z. Z. Wu, Z. Ren, Zhangli Ren, Zheyi Sha, Zhe Fu, Xu Zhen, Zhen Huang, Zhenda Zhang, Zhengyan Xie, Zhewen Zhang, Zhibin Hao, Z. Gou, Zhiqiang Ma, Zhihong Yan, Zhipeng Shao, Zhiyu Xu, Zhongyu Wu, Zhuoshu Zhang, Zihui Li, Zijia Gu, Zhu Zijun, Zilin Liu, Ziwei Li, Xie · 发表于：Annual Meeting of the Association for Computational Linguistics · 年份：2025 · DOI：10.18653/v1/2025.acl-long.766 · 被引用次数：33 · 研究领域：Computer Science

We introduce FinanceReasoning, a novel benchmark designed to evaluate the reasoning capabilities of large reasoning models (LRMs) in financial numerical reasoning problems. Compared to existing benchmarks, our work provides three key advancements. (1) Credibility: We update 15.6% of the questions from four public datasets, annotating 908 new questions with detailed Python solutions and rigorously refining evaluation standards. This enables an accurate assessment of the reasoning improvements of LRMs. (2) Comprehensiveness: FinanceReasoning covers 67.8% of financial concepts and formulas, significantly surpassing existing datasets. Additionally, we construct 3,133 Python-formatted functions, which enhances LRMs'financial reasoning capabilities through refined knowledge (e.g., 83.2% $\rightarrow$ 91.6% for GPT-4o). (3) Challenge: Models are required to apply multiple financial formulas for precise numerical reasoning on 238 Hard problems. The best-performing model (i.e., OpenAI o1 with PoT) achieves 89.1% accuracy, yet LRMs still face challenges in numerical precision. We demonstrate that combining Reasoner and Programmer models can effectively enhance LRMs'performance (e.g., 83.2% $\rightarrow$ 87.8% for DeepSeek-R1). Our work paves the way for future research on evaluating and improving LRMs in domain-specific complex reasoning tasks.