Large-scale human predictions of the replicability of published social and behavioural science papers – a multi- study analysis

作者：Fallon Mody, David P. Wilkinson, Hannah Fraser, Fiona Fidler, Michaël Bishop, Mark A. Burgman, Martin Bush, Yiling Chen, Anna Dreber, Brandon Goldfedder, Michael Gordon, Elliot Gould, Charles T. Gray, Rebecca Groenewegen, Daniel G. Hamilton, Anca M. Hanea, Andrew Head, Victoria Hemming, Felix Holzmeister, Magnus Johannesson, Alexandru Marcoci, Rose E. O'Dea, Ross Pearson, Libby Rumpff, Felix Singleton Thorn, Eden T. Smith, Louisa Tran, Peter A. Vesk, Juntao Wang, Aaron Willcox, Bonnie C. Wintle, Charles Twardy, Thomas Pfeiffer · 年份：2026 · DOI：10.31222/osf.io/vgyed_v1 · 被引用次数：2 · 研究领域：Meta-analysis and systematic reviews、Forecasting Techniques and Applications、Computational and Text Analysis Methods

This paper reports two approaches to forecasting replicability for a corpus of 3000 published social science papers, using large-scale human assessments. Replication Markets used incentivized surveys and prediction markets where participants traded assets linked to replication outcomes and market prices were interpreted as replication predictions. The repliCATS project used a structured group deliberation protocol, including interactive discussion, and mathematically aggregated forecasts to generate replication predictions. Accuracy for both approaches was validated against a subset of (n=37) independent, high-power replication studies. The predictive accuracy (median [range]) achieved for AUC by Replication Markets was 0.76 [0.72-0.82], and 0.76 [0.68-0.81] by repliCATS. Replication Markets achieved a classification accuracy of 73% [68-76%] and repliCATS achieved 68% [59-78%]. These results place the performance of both teams within the accuracy range achieved in prior replication forecasting studies. We conclude that informative forecasts can be elicited by both methods, but there are trade-offs between scale and accuracy.