Benchmarking cell type and gene set annotation by large language models with AnnDictionary

作者：George Crowley, Robert C. Jones, Mark A. Krasnow, Angela Oliveira Pisco, Julia Salzman, Nir Yosef, Siyu He, Madhav Mantri, J. L. Corrales Aguirre, R.C. Garner, Sal Guerrero, William Harper, Resham Irfan, Sophia Mahfouz, R. Ponnusamy, Bhavani A. Sanagavarapu, Ahmad Salehi, Ivan Sampson, Chloe Tang, Alan G. Cheng, James M. Gardner, Burnett S. Kelly, Thurman Slone, Zifa Wang, A.D. Choudhury, Sheela Crasta, Chen Dong, Marcus L. Forst, Douglas E. Henze, Jaeyoon Lee, Maurizio Morri, Serena Y. Tan, Sevahn K. Vorperian, Lynn Yang, Marcela Alcántara-Hernádez, Julian Berg, Dhruv Bhatt, Sara Billings, Andres Gottfried‐Blackmore, Jamie Bozeman, Simon Bucher, Elisa B. Caffrey, Amber Casillas, Rebecca Chen, Matthew Choi, Rebecca N. Culver, Ivana Cvijović, Ke Ding, Hala Shakib Dhowre, Dong Hua, Kenneth Donaville, Lauren Duan, Xiaochen Fan, Mariko H. Foecke, Francisco X. Galdos, Eliza A. Gaylord, Karen Gabriel Gonzales, William R. Goodyer, Michelle Griffin, Yuchao Gu, Shuo Han, Jun He, Paul V. Heinrich, Rebeca Arroyo Hornero, Keliana Hui, Juan C. Irwin, SoRi Jang, Annie Jensen, Saswati Karmakar, Jengmin Kang, Hailey Kang, Soochi Kim, Stewart J. Kim, William Kong, Mallory A. Laboulaye, Daniel Lee, Gyehyun Lee, Elise Lelou, Anping Li, Baoxiang Li, Wan-Jin Lu, Hayley M. Raquer-McKay, Elvira Mennillo, Lindsay S. Moore, Elena Montauti, Karim Mrouj, Shravani Mukherjee, Patrick Neuhöfer, Sunny Nguyen, Honor Paine, Jennifer Parker, Julia H. Pham, Kiet T. Phong, Pratima Prabala, Zhen Qi, Joshua Quintanilla, I Rusu, Ali Reza Rais Sadati, Bronwyn Scott, David Seong, Ho‐Su Sin, Hanbing Song, Bikem Soyur, Sean P. Spencer, Varun Ramanan Subramaniam, Michael Swift, Aditi Swarup, Gregory L. Szot, Aris Taychameekiatchai, Emily Trimm, Stefan Veizades, Sivakamasundari Vijayakumar, Kim Chi Vo, Tian Wang, Ting-Hsuan Wu, Yinghua Xie, William Yue, Zue Zhang, Angela M. Detweiler, Honey Mekonen, Norma Neff, Sheryl Paul, Amanda Seng, Jia Yan, Deana R.C. Colburg, Balint Forgo, Luca Ghita, Frank McCarthy, Aditi Agrawal, Alina Isakova, Kavita Murthy, Alexandra Psaltis, Wenfei Sun, Kyle Awayan, Pierre Boyeau, Robrecht Cannoodt, Leah C. Dorman, Samuel D’Souza, Can Ergen, Justin Hong, Harper Hua, Erin McGeever, Antoine de Morrée, Luise A. Seeker, Alexander J. Tarashansky, Astrid Gillich, Taha A. Jan, Angela H. Ling, Abhishek Murti, Nikita Sajai, Ryan M. Samuel, Juliane Winkler, Steven E. Artandi, Philip A. Beachy, Mike F. Clarke, Zev J. Gartner, Linda C. Giudice, Franklin W. Huang, Juliana Idoyaga, Michael G. Kattah, Christin S. Kuo, Diana J. Laird, Michael T. Longaker, Patricia K. Nguyen, David Y. Oh, Thomas A. Rando, Kristy Red-Horse, Bruce Wang, Albert Y. Wu, Sean M. Wu, Bo Yu, James Zou, Stephen R. Quake · 发表于：Nature Communications · 年份：2025 · DOI：10.1038/s41467-025-64511-x · 被引用次数：5 · 研究领域：Machine Learning and Algorithms、Machine Learning in Bioinformatics、Biomedical Text Mining and Ontologies

We develop an open-source package called AnnDictionary to facilitate the parallel, independent analysis of multiple anndata. AnnDictionary is built on top of LangChain and AnnData and supports all common large language model (LLM) providers. AnnDictionary only requires 1 line of code to configure or switch the LLM backend and it contains numerous multithreading optimizations to support the analysis of many anndata and large anndata. We use AnnDictionary to perform the first benchmarking study of all major LLMs at de novo cell-type annotation. LLMs vary greatly in absolute agreement with manual annotation based on model size. Inter-LLM agreement also varies with model size. We find that LLM annotation of most major cell types to be more than 80-90% accurate, and will maintain a leaderboard of LLM cell type annotation. Furthermore, we benchmark these LLMs at functional annotation of gene sets, and find that Claude 3.5 Sonnet recovers close matches of functional gene set annotations in over 80% of test sets. Cell type labelling in single-cell datasets remains a major bottleneck. Here, the authors present AnnDictionary, an open-source toolkit that enables atlas-scale analysis and provides the first benchmark of LLMs for de novo cell type annotation from marker genes, showing high accuracy at low cost.