Tuesday, January 21, 2014

AER: Replications?

Maniadis, Tufano, and List build on "Why Most Research Findings are False" in their new January 2014 AER article (older, ungated draft here) promoting the virtues of replicating experiments to an economics audience.

In particular, through a simple model they formalize that the probability that a declaration of a research finding (made upon reaching statistical significance) is true...
    1) declines the more surprising the result,
    2) is larger for larger samples,
    3) is decreasing in the number of researchers examining a phenomenon, and
    4) is decreasing in research bias.

As an example, they replicate a famous anchoring experiment (Ariely, Loewenstein, and Prelec QJE 2003)*. The replication yields much weaker results than the original study.

I am happy the AER decided to publish an article like this, but I wonder where the article would have been published if the authors had ONLY included the replication of the anchoring experiment? And what if the replication confirmed the original study, instead of questioning it? I bet it wouldn't be published in the AER or QJE.

Until replications are vetted and visible -- even the really boring ones that run the same experiment with different subjects and get qualitatively the same results -- replicating experiments (i.e. the lack of it) will be a problem.

One way the situation could be improved is for a journal, say the AER, to have a Replication section (even if it's online only) specifically for publishing replications of AER papers. Call it AER: Replications. Having a separate section or sister journal would serve two ends:
   1) There would be a place to publish replications and for people to look for them.
   2) Replications would be separate from "original" work. This way, replications aren't "taking the spot" of a "more interesting" or "goundbreaking" paper and universities could easily identify replications, as well. So for calculating tenure, for example, AER: Replications would be treated differently than AER. There is a precedent for this: AER: Papers and Proceedings is usually treated differently than AER.

Related Post:
How Science Goes Wrong

* An aside: There's a weird footnote in the MTL paper mentioning the data from ALP's 2003 experiment are "lost". For such an important, recent study, it's weird to me that the data could be lost.

Thursday, January 2, 2014

The End... of Light Bulbs (TFS, Part 12)

I am reading Thinking, Fast and Slow, by Daniel Kahneman. In this series I will summarize key parts of the book and supply some comments and reflections on the material.

Part V: Two Selves
Chapters 35-38, Conclusion

Summary:

In addition to the "two systems" (fast System 1 and slow System 2), we have "two selves", the remembering self and the experiencing self. The experiencing self answers the question, "how is it, now?" The remembering self answers the question, "how was it, on the whole?" and usually makes decisions about repeating experiences. These decisions may conflict with what the experiencing self would want. The conflict comes from the remembering self's desire to "tell a good story" and "rating" experiences based mostly on the worst moment of the experience and what happened at the end of the experience. This leads to duration neglect.  

Kahneman reminds us that "how happy are you?", "how was it, on the whole?", and related questions are very difficult to answer. System 1 could be using biased heuristics to answer different, easier questions, instead, so it may be difficult to draw conclusions from this research.

Nevertheless, Kahneman issues some of his big takeaways from this literature:
  • "It is only a slight exaggeration to say that happiness is the experience of spending time with people you love and who love you." 
  • "Can money buy happiness? The conclusion is that being poor makes one miserable..." 
  • "The goals people set for themselves are so important to what they do and how they feel about it that an exclusive focus on experienced well-being is not tenable. We cannot hold a concept of well-being that ignores what people want. On the other hand, it is also true that a concept of well-being that ignores how people feel as they live and focuses only on how they feel when they think about their life is also untenable. We must accept the complexities of a hybrid view, in which the well-being of both selves is considered."
 Kahneman spends the conclusion criticising the "ideology" of "Chicago economics" (he calls out Milton Friedman and Gary Becker specifically), and strongly endorsing libertarian paternalism, behavioral economics, and the agenda outlined in Nudge.

THE END.

My Thoughts:

I get confused by what people mean when they contrast "Chicago school" and the work of Thaler (of Nudge fame). He's also a Chicago economist. Oh well.

I support some aspects of libertarian paternalism (e.g. when picking a default option is unavoidable, you should pick one that is expected to be best for most people to ease the decision making process), but I don't like the term nudge or thinking about nudges as their own thing. "Nudges" are really just additional small regulations or rules. Instead of thinking about nudges, we should be asking, what is the right amount of regulation? or What is the right rule? Sometimes more is needed, sometimes less. Sometimes regulations and rules should just be different. How strong can a nudge be before it's not a nudge, anyway? "Nudge" seems more like PR than actual policy to me. It distracts from the real issues at hand: the goal and effects of regulations and rules. It's especially distracting when people have different goals and preferences regarding policy. Who can be against something labeled a mere nudge? No one!

Speaking of nudges, the light bulb ban went into full effect yesterday. Supporters call it a nudge because it encourages people to conserve electricity and over time and if people aren't picking the more expensive bulb, they are obviously making a terrible choice harming themselves, their family, and the entire world. To me, this is not a nudge because it reduces choice. There is no opt out. I guess there is an opt out, but the opt out is too expensive: you could have stocked up on regular bulbs before the ban that last you a lifetime.

The light bulb ban is bad. Yes, people's electric bills will be lower with the new bulbs (assuming lifetime estimates are accurate), but you have to pay a higher upfront cost, the new bulbs look different, they don't always fit the same fixtures (which are expensive to replace), and they give off different types of light (maybe I just like certain types of light). And these are real costs to this ban, no matter what these people say. But, you say, these costs are really low! Really, how much do you care about having a nice yellow glow and paying less up front, anyway? Well, enough to not want to switch light bulbs!

Maybe people (me?) don't know about how great these bulbs are? But there is information on the front of the box tells them they will save money, how bright they are, the color, and the price. Maybe people don't know math or how to read? If that's the case, light bulb bans are not addressing the key problem here.

Even if people are stupid and that's okay because the government can force them to do what's in their best interest anyway (paternalism), what is the problem the ban solves? If levels of energy consumed is the concern (maybe because of global warming?), it's better to tax energy usage directly or use quotas or cap and trade to reduce overall consumption. Maybe there are cheaper or more preferred alternatives to lower energy usage, like turning off the lights in the house or turning off their computer or turning down the A/C (which actually uses orders of magnitude more power than a light bulb). For me, light bulbs are practically nothing in the grand scheme of energy usage (look at your electric bill next month and do the calculation for yourself; running one 100W bulb for 100 hours is 10 KWH and costs about one dollar and ten cents in Illinois). Let people choose! But I guess if people are dumb enough to buy the wrong bulbs they are probably dumb enough to choose to reduce electricity consumption on something that hurts them a lot rather than a little?

If you are concerned about the cost of electricity (and the poor's access to it), then why not consider expanding supply? Possibly by increasing offshore drilling, drilling in Alaska, or building more pipelines (all are restricted by federal regulatory bodies, keeping the cost of energy artificially high).

Supporting nuclear power and a place in Nevada to safely store the waste would address both global warming concerns of increasing energy supply (the "smoke" coming out of nuclear power plants is water vapor) and lower the cost of electricity by increasing supply. Again, federal regulations and indecision about where to store waste prevent the building of more nuclear plants.

The light bulb ban is more of a nuisance than an actual solution to an important problem.
 
Food for Thought

1) Kahneman likes poking fun at the standard modeling assumption that preferences are consistent and unchanging. However, he takes subjects' reports on "fairness" of outcomes, assumes stability, and derives policy implications from them. Can humans have inconsistent preferences, but consistent notions of fairness?

2) "Freedom has a cost, which is borne by individuals who make bad choices, and by a society that feels obligated to help them.... For adherents of [the Chicago school], freedom is free of charge."

Do you agree? How responsible should people be for themselves? If you "feel obligated to help" someone who is making a personal decision that physically only affects him, then does that decision harm you? Does that give you a right to stop him from making that decision? What if lots of other people "feel obligated to help," too? Does it matter if the action you take actually helps (or hurts) him?

3) Alice: "It is common knowledge that smoking is addictive, causes health problems, and makes one feel good. I will take up smoking." Bob: "It is common knowledge that smoking is addictive, causes health problems, and makes one feel good. I will prevent you from smoking." Cass: "It is common knowledge that smoking is addictive, causes health problems, and makes one feel good. I will make it hard for you to find cigarettes and make them more expensive so that you are nudged towards not smoking."

If smoking harms others (e.g. second hand smoke), how should that affect the community's decision on whether to ban it or tax it?

4) Could forcing people to use energy efficient light bulbs actually lead people to use more energy on light? (Hint: the answer is yes. Why?)