Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

作者：Elliot Gould, Hannah Fraser, Timothy Parker, Shinichi Nakagawa, Simon C. Griffith, Peter A. Vesk, Fiona Fidler, Daniel G. Hamilton, Robin N. Abbey-Lee, Jessica K. Abbott, Luis Alejandro Aguirre, Carles Alcaráz, Irith Aloni, Drew Altschul, Kunal Arekar, Jeff W. Atkins, Joe Atkinson, Christopher M. Baker, Meghan Barrett, Kristian Bell, Suleiman Kehinde Bello, Iván Beltrán, Bernd J. Berauer, Michael G. Bertram, Peter D. Billman, Charlie Blake, S. F. Blake, Louis Bliard, Andrea Bonisoli‐Alquati, Timothée Bonnet, Camille Bordes, Aneesh P. H. Bose, Thomas Botterill‐James, Melissa Anna Boyd, Sarah A. Boyle, Tom Bradfer‐Lawrence, Jennifer Bradham, Jack A. Brand, Martin Brengdahl, Martin Bulla, Luc F. Bussière, Ettore Camerlenghi, Sara E. Campbell, Leonardo Leite Ferraz de Campos, Anthony Caravaggi, Pedro Cardoso, Charles J. W. Carroll, Therese A. Catanach, Xuan Chen, Heung Ying Janet Chik, Emily S. Choy, Alec Philip Christie, Angela Chuang, Amanda J. Chunco, Bethany L. Clark, Andrea Contina, Garth A. Covernton, Murray P. Cox, Kimberly Cressman, Marco Crotti, Connor D. Crouch, Pietro B. D’Amelio, Alexandra A. de Sousa, Timm F. Döbert, Ralph Dobler, A. Dobson, Tim S. Doherty, Szymon M. Drobniak, Alexandra G. Duffy, Alison B. Duncan, Robert P. Dunn, Jamie Dunning, Trishna Dutta, Luke Eberhart-Hertel, Jared A. Elmore, Mahmoud Medhat Elsherif, Holly M. English, David C. Ensminger, Uli Ernst, Stephen M. Ferguson, Esteban Fernández‐Juricic, Thalita Ferreira‐Arruda, John Fieberg, Elizabeth A. Finch, Evan A. Fiorenza, David N. Fisher, Amélie Fontaine, Wolfgang Forstmeier, Yoan Fourcade, Graham S. Frank, Cathryn A. Freund, Eduardo Fuentes‐Lillo, Sara Gandy, Dustin Gannon, Ana I. García‐Cervigón, Alexis Garretson, Xuezhen Ge, William L. Geary, Charly Géron, Marc Gilles, Antje Girndt, Daniel Gliksman, Harrison Goldspiel, Dylan Gomes, Megan Good, Sarah Goslee, J. Stephen Gosnell, Eliza M. Grames, Paolo Gratton, Nicholas M. Grebe, Skye M. Greenler, Maaike Griffioen, Daniel M. Griffith, Frances J. Griffith, Jake J. Grossman, Ali Güncan, Stef Haesen, James G. Hagan, Heather A. Hager, Jonathan P. Harris, Natasha D. Harrison, Sarah Hasnain, Justin C. Havird, Andrew Heaton, María Laura Herrera-Chaustre, Tanner J. Howard, Bin‐Yan Hsu, Fabiola Iannarilli, Esperanza C. Iranzo, Erik N. K. Iverson, Saheed Olaide Jimoh, Douglas H. Johnson, Martin Johnsson, Jesse Jorna, Tommaso Jucker, Martin Jung, Ineta Kačergytė, Oliver Kaltz, Alison Ke, Clint D. Kelly, Katharine Keogan, Friedrich W. Keppeler, Alexander K. Killion, Dongmin Kim, David P. Kochan, Peter Korsten, Shan Kothari, Jonas Kuppler, Jillian M. Kusch, Malgorzata Lagisz, Kristen Lalla, Daniel J. Larkin, Courtney L. Larson, Katherine S. Lauck, M. Elise Lauterbur, Alan Law, Don‐Jean Léandri‐Breton, Jonas J. Lembrechts, Kiara L. L'Herpiniere, Eva Lievens, Daniela Oliveira de Lima, Shane Lindsay, Martin Luquet, Ross MacLeod, Kirsty H. Macphie, Kit Magellan, Magdalena M. Mair, Lisa Malm, Stefano Mammola, Caitlin P. Mandeville, Michael Manhart, Laura Milena Manrique‐Garzón, Elina Mäntylä, Philippe Marchand, Benjamin Michael Marshall, Charles A. Martin, Dominic A. Martin, Jake M. Martin, April Robin Martinig, Erin S. McCallum, Mark McCauley, Sabrina M. McNew, Scott J. Meiners, Thomas Merkling, Marcus Michelangeli, María Moirón, Bruno Moreira, Jennifer A. Mortensen, Benjamin Mos, Taofeek O. Muraina, Penelope Murphy, Luca Nelli, Petri T. Niemelä, Josh Nightingale, Gustav Nilsonne, Sergio Nolazco, Sabine S. Nooten, Jessie Lanterman Novotny, Agnes B. Olin, Chris L. Organ, Kate L. Ostevik, Facundo X. Palacio, Matthieu Paquet, Darren J. Parker, David J. Pascall, Valerie J. Pasquarella, John Harold Paterson, Ana Payo‐Payo, Karen Pedersen, Grégoire Perez, Kayla I. Perry, Patrice Pottier, Michael J. Proulx, Raphaël Proulx, Jessica L. Pruett, Veronarindra Ramananjato, Finaritra Randimbiarison, Onja H. Razafindratsima, Diana J. Rennison, Federico Riva, Sepand Riyahi, Michael J. Roast, Felipe Pereira da Rocha, Dominique G. Roche, Cristian Román‐Palacios, Michael S. Rosenberg, Jessica Ross, Freya E. Rowland, Deusdedith M. Rugemalila, Avery L. Russell, Suvi Ruuskanen, Patrick Saccone, Asaf Sadeh, Stephen M. Salazar, Kris Sales, Pablo Salmón, Alfredo Sánchez‐Tójar, Leticia Santos, Francesca Santostefano, Hayden T. Schilling, Marcus Schmidt, Tim Schmoll, Adam C. Schneider, Allie E. Schrock, Julia Schroeder, Nicolas Schtickzelle, Nick L. Schultz, Drew A. Scott, Michael P. Scroggie, Julie Teresa Shapiro, Nitika Sharma, Caroline L. Shearer, Diego Simón, Michael I. Sitvarin, Fabrício Luiz Skupien, Heather L. Slinn, Grania Smith, Jeremy A. Smith, Rahel Sollmann, Kaitlin Stack Whitney, Shannon Still, Erica F. Stuber, Guy F. Sutton, Ben Swallow, Conor C. Taff, Elina Takola, Andrew J. Tanentzap, Rocío Tarjuelo, Richard J. Telford, Christopher J. Thawley, Hugo Thierry, Jacqueline Thomson, Svenja Tidau, Emily M. Tompkins, Claire M. Tortorelli, Andrew Trlica, Biz R. Turnell, Lara Urban, Stijn Van de Vondel, Jessica E. M. van der Wal, Jens Van Eeckhoven, Francis van Oordt, K. Michelle Vanderwel, Mark C. Vanderwel, Karen J. Vanderwolf, Juliana Vélez, Diana Carolina Vergara‐Flórez, Brian C. Verrelli, Marcus Vinícius Vieira, Nora Villamil, Valerio Vitali, Julien Vollering, Jeffrey A. Walker, Xanthe J. Walker, Jonathan A. Walter, Paweł Waryszak, Ryan J. Weaver, Ronja E. M. Wedegärtner, Daniel L. Weller, Shannon Whelan, Rachel L. White, David W. Wolfson, Andrew Wood, Scott W. Yanco, Jian D. L. Yen, Casey Youngflesh, Giacomo Zilio, Cédric Zimmer, Gregory M. Zimmerman, Rachel A. Zitomer · 发表于：BMC Biology · 年份：2025 · DOI：10.1186/s12915-024-02101-x · 被引用次数：52 · 研究领域：Species Distribution and Climate Change、Data Analysis with R、Meta-analysis and systematic reviews

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus datase...