Friday, June 20, 2014

Is There a Crisis of False Negatives in Psychology?

There are many reasons why a false negative could occur, including these:
  • Replications might be conducted by researchers who are inexperienced or lack expertise, either in general or in the particular area they are trying to replicate.
  • As has been well documented, researchers are human and can act in ways that make them more likely to confirm a hypothesis, resulting in p-hacking. But replicators are human too, and if their hypothesis is that an effect will not replicate, they too can act in ways that increase the likelihood of obtaining that outcome—a practice we might call p-squashing. For example, it would be relatively easy to take an independent variable that had a significant effect in the laboratory, translate it into an on-line study that delivers the manipulation in a much weaker fashion, and then run hundreds of participants, resulting in a null effect. Adding such a study to a meta-analysis could cancel out positive findings from several smaller studies because of its very large sample size, resulting in meta p-squashing.
  • As others have noted (e.g., Stroebe & Strack, 2013), a direct replication could fail because it was conducted in a different context or with a different population, and as a result did not manipulate the psychological construct in the same manner as did the original study.
Do I have evidence that many of the studies that have been done as part of the current replication movement have been plagued by the above problems? Well, not much, though I suggest that the evidence is equally weak that false positives are rampant. One might even argue that there is just as much evidence that we have a crisis of false negatives as we do a crisis of false positives.

- More Here

No comments: