Replication by Plebiscite: Examining the Replicability of Online Experiments Selected by a Decision Market

Abstract: We implemented a decision market on 41 systematically selected MTurk social science experiments published in PNAS between 2015 and 2018. Social scientists (n = 162) traded on the outcome of direct replications, knowing that the 12 studies with the lowest and the 12 with the highest final market prices would be selected for replication, along with two randomly selected studies. The replication rate was 83% for the top-12 and 33% for the bottom-12 group. The correlation between decision market prices and replication outcomes was 0.505, indicating some potential for decision markets to select studies for replication. Overall, 54% of the studies were successfully replicated, with replication effect sizes averaging 45% of the original effect sizes. These results suggest that the replicability of MTurk experiments is comparable to that of previous systematic replication projects involving laboratory experiments. All study aspects were preregistered, and we strictly adhered to our pre-analysis plan.

Recommended citation: Holzmeister, F., Johannesson, M., Camerer, C. F., Chen, Y., Ho, T-H., Hoogeveen, S., Huber, J., Imai, N., Imai, T., Jin, L., Kirchler, M., Ly, A., Mandl, B., Manfredi, D., Nave, G., Nosek, B. A., Pfeiffer, T., Sarafoglou, A., Schwaiger, R., Wagenmakers, E-J., Waldén, V., Dreber, A. (2024). Replication by Plebiscite: Examining the Replicability of Online Experiments Selected by a Decision Market, Forthcoming: Nature Human Behavior.
Download Paper