Reply to Krefeld-Schwalb et al.: Measuring population heterogeneity requires upholding good scientific practice

作者：Felix Holzmeister, Magnus Johannesson, Robert Böhm, Anna Dreber, Jürgen Huber, Michael Kirchler · 发表于：Proceedings of the National Academy of Sciences · 年份：2025 · DOI：10.1073/pnas.2426330122 · 被引用次数：4 · 研究领域：Climate Change Communication and Perception、Mental Health Research Topics、Behavioral Health and Interventions

We appreciate Krefeld-Schwalb et al.'s (KHJ henceforth) ( 1 ) interest in our study ( 2 ) and critical discourse spurred by their commentary.We largely agree with KHJ's theoretical arguments, which closely relate to the caveats discussed in our manuscript.Particularly, we acknowledge that the reviewed multilab studies are typically based on samples from WEIRD countries, which may entail lower heterogeneity than in other settings ( 3 , 4 )."Put differently, our comparatively low estimates of population heterogeneity might be subject to population heterogeneity itself."( 2 ).However, we express reservations about KHJ's empirical claims about the magnitude of population heterogeneity, drawing on Krefeld-Schwalb et al. (KSJ henceforth) ( 5 ).It seems that KSJ intentionally studied paradigms expected to yield large meta-analytic effect sizes and "employed purposive variation of the sampling frame" ( 1 ) to enhance heterogeneity.Olsson-Collentine et al. ( 6) provide evidence for a correlation between effect sizes and heterogeneity.Study 1 in KSJ documents effect size estimates of four paradigms across ten online samples and one laboratory sample.KHJ's estimates of H , ranging from 1.7 to 9.6, suggest that population heterogeneity is markedly larger than the average level of population heterogeneity observed in our sample.However, both KSJ and KHJ fail to report estimates for a fifth preregistered paradigm embedded in KSJ's study-the "local warming" effect-and omit preregistered ana...