Saturday, April 27, 2013

Questionable Studies

Academic publication is a dialog, and bad science happens for a lot of different reasons, but it seems like this month, certain papers with problems have become unusually popularized (e.g. given full articles in the WSJ and other newspapers).

Reinhart and Rogoff ask: How does debt relate to growth?

This is an important question, and the way they framed their results (implying a debt/GDP cutoff where growth becomes hampered), got a lot of press when published 3 years ago and was cited as the basis of a lot of austerity measures. Sadly, there are mistakes. In particular, there was an Excel coding error. The authors have also been accused of selectively omitting, or cherry picking, their data, and implying that debt causes slow growth when the causation could easily go the other way, that is, making a correlation implies causation error.

In their response to the criticism, R&R admit the Excel error and strongly deny the cherry picking and causation error. To be fair, it was more of the press and interested governmental parties that were more than happy to claim this paper proved a causation.

Here's a "scorecard" of the saga with a description of how the results change, links to the original paper, the criticism, the response, and other relevant things. Note that it seems like the error may have been pointed out in a way to maximize press coverage and that R&R is just one of the many papers on the debt/growth question.

It's not just macroeconomics, biology has some issues, too.

Sharov and Gordon claim: Life on Earth originated "elsewhere." 

I am just glad this paper is on arxiv and hasn't been picked up yet by a journal (I don't think).  Here's figure 1:



Here's a great description of the ridiculousness of it. The WSJ correctly pointed out they use only 5 data points -- a severe lack of data that bests even macro papers. However, as Myers points out, they also cherry pick those data points, and abuse extrapolation.


-----
References:

Here's the Reinhart and Rogoff paper.

Here's the Sharov and Gordon paper.


No comments:

Post a Comment