Tuesday, January 21, 2014

AER: Replications?

Maniadis, Tufano, and List build on "Why Most Research Findings are False" in their new January 2014 AER article (older, ungated draft here) promoting the virtues of replicating experiments to an economics audience.

In particular, through a simple model they formalize that the probability that a declaration of a research finding (made upon reaching statistical significance) is true...
    1) declines the more surprising the result,
    2) is larger for larger samples,
    3) is decreasing in the number of researchers examining a phenomenon, and
    4) is decreasing in research bias.

As an example, they replicate a famous anchoring experiment (Ariely, Loewenstein, and Prelec QJE 2003)*. The replication yields much weaker results than the original study.

I am happy the AER decided to publish an article like this, but I wonder where the article would have been published if the authors had ONLY included the replication of the anchoring experiment? And what if the replication confirmed the original study, instead of questioning it? I bet it wouldn't be published in the AER or QJE.

Until replications are vetted and visible -- even the really boring ones that run the same experiment with different subjects and get qualitatively the same results -- replicating experiments (i.e. the lack of it) will be a problem.

One way the situation could be improved is for a journal, say the AER, to have a Replication section (even if it's online only) specifically for publishing replications of AER papers. Call it AER: Replications. Having a separate section or sister journal would serve two ends:
   1) There would be a place to publish replications and for people to look for them.
   2) Replications would be separate from "original" work. This way, replications aren't "taking the spot" of a "more interesting" or "goundbreaking" paper and universities could easily identify replications, as well. So for calculating tenure, for example, AER: Replications would be treated differently than AER. There is a precedent for this: AER: Papers and Proceedings is usually treated differently than AER.

Related Post:
How Science Goes Wrong

* An aside: There's a weird footnote in the MTL paper mentioning the data from ALP's 2003 experiment are "lost". For such an important, recent study, it's weird to me that the data could be lost.

No comments:

Post a Comment