Saturday, November 2, 2013

How Science Goes Wrong

The Economist did a nice cover piece last week on How Science Goes Wrong. It is worth reading in it's entirety. Here's the leader and here's the cover story.

A major problem in science is the lack of incentives to replicate others' research. This is the thesis of the piece.

A problem that they don't mention directly, but alludes to, is that bad studies keep getting cited as truth even though they've been retracted, debunked, or otherwise discredited. EVEN IN THEIR OWN FIELD. Angrist and Krueger (1991) I AM LOOKING AT YOU. You would think that once someone showed their results could be qualitatively replicated using random data the paper (and technique of using quarter of birth as a valid instrument in the setting) would be relegated to the dustbin. You would be wrong. David, I feel your pain.

There are numerous examples underlying this thesis cited in the Economist, which should be concerning to anyone who values scientific research and its (subsidized) place in the modern economy.
A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 “landmark” studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties.
... nine separate experiments had not managed to reproduce the results of a famous study from 1998 purporting to show that thinking about a professor before taking an intelligence test leads to a higher score than imagining a football hooligan. 
Why the downturn in verification and replication?
The obligation to "publish or perish" has come to rule over academic life.... Careerism also encourages exaggeration and the cherry-picking of results.
The cover story does a good job of going through the statistics of how most published research (even when done right and in good faith) could be wrong. And it's not all in good faith. Because replications and reviews are not valued highly, and moderate or negative results are not usually interesting. It's important not only to do good work, but capture the interest of your fellow scientists when publishing. This leads to incentives to distort or exaggerate your work. If it's controversial, even better -- you'll get cited. Truth matters less than sparking debate. Journals publish successes, not failures.

And those success are rarely checked as carefully as they need to be by peer review once papers make it to the journal stage. Editors and reviewers need to spend time writing their own papers, not reading someone else's! Usually, the paper reviews get dumped to grad students as busy work, further reducing the quality of the review.
When a prominent medical journal ran research past other experts in the field, it found that most of the reviewers failed to spot mistakes it had deliberately inserted into papers, even after being told they were being tested.
... John Bohannon, a biologist at Harvard, recently submitted a pseudonymous paper on the effects of a chemical derived from lichen on cancer cells to 304 journals describing themselves as using peer review. An unusual move; but it was an unusual paper, concocted wholesale and stuffed with clangers in study design, analysis and interpretation of results. Receiving this dog’s dinner from a fictitious researcher at a made up university, 157 of the journals accepted it for publication.
 How to Fix the Problem?

One thing the Economist notes, which I fully agree with, is that applied scientists need to understand statistics better. Also, scientists should actually follow the scientific method. Unfortunately, something like this usually happens instead -- if you're lucky! Here's more of their recommendation:
Journals should allocate space for “uninteresting” work, and grant-givers should set aside money to pay for it. Peer review should be tightened—or perhaps dispensed with altogether, in favour of post-publication evaluation in the form of appended comments. That system has worked well in recent years in physics and mathematics. Lastly, policymakers should ensure that institutions using public money also respect the rules.

Why Does this all Matter, Anyway?
The governments of the OECD, a club of mostly rich countries, spent $59 billion on biomedical research in 2012, nearly double the figure in 2000. One of the justifications for this is that basic-science results provided by governments form the basis for private drug-development work. If companies cannot rely on academic research, that reasoning breaks down. When an official at America’s National Institutes of Health (NIH) reckons, despairingly, that researchers would find it hard to reproduce at least three-quarters of all published biomedical findings, the public part of the process seems to have failed. [emphasis added.]
And that's the problem in a nutshell. Basic research is funded publicly because it is a public good -- so every company ever doesn't have to reinvent the wheel -- that's wasteful. But if that basic research is mostly wrong, and every company that wants to use the result has to independently verify it, why are we funding basic research publicly, anyway?

Science has to be better at establishing actual Truth. A threat of retracting funding should be the stick that whips science into shape.

No comments:

Post a Comment