Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming

Abstract

According to a recent meta-analysis, religious priming has a positive effect on prosocial behavior (Shariff et al., 2015). We first argue that this meta-analysis suffers from a number of methodological shortcomings that limit the conclusions that can be drawn about the potential benefits of religious priming. Next we present a re-analysis of the religious priming data using two different meta-analytic techniques. A Precision-Effect Testing - Precision-Effect-Estimate with Standard Error (PET-PEESE) meta-analysis suggests that the effect of religious priming is driven solely by publication bias. In contrast, an analysis using Bayesian bias correction suggests the presence of a religious priming effect, even after controlling for publication bias. These contradictory statistical results demonstrate that meta-analytic techniques alone may not be sufficiently robust to firmly establish the presence or absence of an effect. We argue that a conclusive resolution of the debate about the effect of religious priming on prosocial behavior - and about theoretically disputed effects more generally - requires a large-scale, preregistered replication project, which we consider to be the sole remedy for the adverse effects of experimenter bias and publication bias.

Citation

Van Elk, M., Matzke, D., Gronau, Q., Guan, M., Vandekerckhove, J., & Wagenmakers, E. (2015). Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming. Frontiers in Psychology, 6, 1365.

Bibtex

@article{van_elk_etal:2015:replications,
    title   = {{M}eta-analyses are no substitute for registered replications: a skeptical perspective on religious priming},
    author  = {Van Elk, Michiel and Matzke, Dora and Gronau, Quentin and Guan, Maime and Vandekerckhove, Joachim and Wagenmakers, Eric-Jan},
    year    = {2015},
    journal = {Frontiers in Psychology},
    volume  = {6},
    pages   = {1365}
}