The Bias is in the Details: An Assessment of Cognitive Bias in LLMs

作者：R. Alexander Knipper, Charles S. Knipper, Kaiqi Zhang, †. ValerieSims, †. ClintBowers, S. Karmaker, Marah Abdin, Jy-oti Aneja, H. Awadalla, Ammar Ahmed Awadallah, Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Singh Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dongkai Chen, Dongkai Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jian-Feng Gao, Mei Gao, Mingcen Gao, Amit Garg, A. Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Jun-heng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Adé Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Y. Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Lidén, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiao-Dong Liu, Chong Luo, Piyush Madan, Alireza Mahmoudzadeh, D. Majercak, Matt Mazzola, Caio César, Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, D. Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Rad-milac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, A. Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, H. Witte, Xiaoxia Wu, Michael Wu, Bin Wyatt, Can Xiao, Jia-Hang Xu, Weijian Xu, Jilong Xu, Sonali Xue, Fan Yadav, Jian-wei Yang, Yi-Fan Yang, Ziyi Yang, Donghan Yang, Lu Yu, Chenruidong Yuan, Cyril Zhang, Jianwen Zhang, Zhang, Li Lyna, Yi Zhang, Yue Zhang, Yunan Zhang, Zhang Xiren, Zhou, Suriya Gunasekar, Michael Harrison, Mojan Javaheripi, James R. Lee, C. C. T. Mendes, Eric Price, Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, Shmargaret Shmitchell, Nuo Chen, Jiqun Liu, Xiaoyu Dong, Qijiong Liu, Tetsuya Sakai, Xiao-Ming Wu. 2024, Ai, Syeda Alexander Choi, Sabrina Akter, J. Singh, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xing Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, A. Liu, Bing Xue, Bing-Li Wang, Bochao Wu, B. Feng, Cheng-Da Lu, Chengqi Zhao, C. Deng, Chenyu Zhang, C. Ruan, Da-Mai Dai, Deli Chen, Dong-Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Hua-Zuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jia-Shi Li, Jiawei Wang, Jing-Chang Chen, Jingyang Yuan, Jun-Jie Qiu, Jun-Long Li, J. Cai, J. Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Qiushi Chen, Ruiqi Du, Ruisong Ge, Ruizhe Zhang, Runji Pan, R. J. Wang, R. L. Chen, Jin, Wanjia Zeng, Wen Zhao, Wenfeng Liu, Wenjun Liang, Wen-Wan Gao, Wentao Yu, W. L. Zhang, W. Xiao, Xiaodong An, Xiaohan Liu, Xiaokang Wang, Xiaotao Chen, Xin Nie, Xin Cheng, Liu Xin, Xingchao Xie, Xin-Yu Liu, Xinyuan Yang, Xuecheng Li, Xuheng Su, Xiangyue Li, Xiaojin Jin, Xiaosha Shen, Chen, A. Demidova, Hanin Atwany, Nour Rabih, J. Echterhoff, Yao Liu, Abeer Alessa, G. Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, J. Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Milli-can, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrit-twieser, A. Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, A. Lazaridou, Orhan Firat, James Molloy, M. Isard, P. Barham, T. Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuan-Zhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem W. Ayoub, Megha Goel, J. Krawczyk, Cosmo Du, E. Chi, Heng-tze Cheng, Eric Ni, Purvi Shah, P. Kane, B. Chan, Manaal Faruqui, A. Severyn, Hanzhao Lin, Yaguang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdieh, Mianna Chen, Pei Sun, Dustin Tran, S. Bagri, Balaji Lakshminarayanan, Jeremiah Liu, András Orbán, Fabian Güra, Hao Zhou, Xiny-ing Song, Aurélien Boffy, H. Ganapathy, Steven Zheng, HyunJeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Majd Emanuel Taropa, Al Merey, M. Baeuml, Zhifeng Chen, L. El, Yujing Zhang, O. Sercinoglu, George Tucker, Enrique Piqueras, M. Krikun, Iain Barr, N. Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Syg-nowski, Alexandre Fréchette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, F. Lebrón, Alban Rrustemi, Natalie Clay, Phil Crone, Tomás Kociský, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam E. Bloniarz, Jack W. Rae, Han Lu, L. Sifre, M. Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, S. Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, R. Chaabouni, Deeni Fatiha, Arun Ahuja, Gau-rav Singh Tomar, Evan Senter, Martin Chad-wick, Ilya Kornakov, Nithya Attaluri, I. Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse, Xavier García, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego De, Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech, D. Reitter, Mianna Chen, Jenny Bren-nan, Clara E. Rivera, Sergey Brin, Shariq Iqbal, G. Surita, Jane Labanowski, Abhishek Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravichandra Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, S. Ganapathy, Smit Sanghavi, A. Kannan, Mingxuan Chang, Axel Stjerngren, J. Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, H. Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Thibault Peter Humphreys, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, V. Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, S. Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, T. Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, A. Chowdhery, D. Das, Dominika Rogozi´nska, V. Nikolaev, P. Sprechmann, Zachary Nado, Lukás Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Christoper A. Welty, Joshua Newlan, Dawei Jia, Miltiadis Allamanis, C. Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu · 发表于：arXiv.org · 年份：2025 · DOI：10.48550/arxiv.2509.22856 · 被引用次数：8 · 研究领域：Computer Science

As Large Language Models (LLMs) are increasingly embedded in real-world decision-making processes, it becomes crucial to examine the extent to which they exhibit cognitive biases. Extensively studied in the field of psychology, cognitive biases appear as systematic distortions commonly observed in human judgments. This paper presents a large-scale evaluation of eight well-established cognitive biases across 45 LLMs, analyzing over 2.8 million LLM responses generated through controlled prompt variations. To achieve this, we introduce a novel evaluation framework based on multiple-choice tasks, hand-curate a dataset of 220 decision scenarios targeting fundamental cognitive biases in collaboration with psychologists, and propose a scalable approach for generating diverse prompts from human-authored scenario templates. Our analysis shows that LLMs exhibit bias-consistent behavior in 17.8-57.3% of instances across a range of judgment and decision-making contexts targeting anchoring, availability, confirmation, framing, interpretation, overattribution, prospect theory, and representativeness biases. We find that both model size and prompt specificity play a significant role on bias susceptibility as follows: larger size (>32B parameters) can reduce bias in 39.5% of cases, while higher prompt detail reduces most biases by up to 14.9%, except in one case (Overattribution), which is exacerbated by up to 8.8%.