Wednesday, November 20, 2013

The Law of Small Numbers? (TFS part 5)

I am reading Thinking, Fast and Slow, by Daniel Kahneman. In this series I will summarize key parts of the book and supply some comments and reflections on the material.

Part II: Heuristics and Biases
Chapter 10: The Law of Small Numbers


Summary: People are bad intuitive statisticians.

Detailed Summary:
  • We incorrectly want to apply the Law of Large Numbers to small numbers as well ("the Law of Small Numbers")
  • We have exaggerated faith in small samples. 
  • We pay more attention to message content than information about its reliability (e.g. sample size, variance).
  • We want to see patterns where there is only randomness.
  • There is no "hot hand" in basketball. 
  • Researchers err when choosing sample size for experiments. "[P]sychologists commonly chose samples so small that they exposed themselves to a 50% risk of failing to confirm their true hypothesis!"

My Thoughts on Chapter 10

First off, a quick quibble: "[P]sychologists commonly chose samples so small that they exposed themselves to a 50% risk of failing to confirm their true hypothesis!" should read "...a 50% risk of failing to reject a false null hypothesis!" 

I also have a quibble with how Kahneman explains our tendency to see patterns where there is only randomness. He claims "causal explanations of chance events are inevitably wrong." This is true, but misleading given the point he is trying to make. If you know the process behind the event is random, then yes, a causal explanation is wrong. But what if you don't know the mechanism? What if you are unsure what is going on? I think what he wants to say is that people are good at finding patterns and bad at accepting the absence of patterns ("randomness"), but I'm not sure. Also, when Kahneman mentions "randomness" and "chance" he almost always is assuming independence of events. Sometimes this assumption matters a lot and he seems to assume it in some places that are questionable. 

Sometimes the assumption of independence of events when events are in fact NOT independent can be disastrous. Case in point: the sub-prime mortgage crisis. All these fancy financial instruments were priced correctly assuming default on mortgages were relatively unrelated. In fact, they were highly correlated. Tim Harford has a fantastic basic description of the crisis and how bad assuming statistical independence can be when it's not true. Read it here. You will never think about eggs or mortgages in quite the same way again.

Given that statistical mismodeling can be so disastrous, is it such a bad thing that people search for patterns and take extra precautions after something unexpected or extremely unlikely happens? Even if it turns out that, often, it's just a fluke?

One last thing -- (and I will come back to this experiment several times in discussing these chapters) people may actually be pretty good intuitive statisticians! Results of experiments that ask difficult questions seem very sensitive to how questions are asked in a laboratory setting -- a lesson I thought we learned when we talked about happiness

Food for Thought:

1) At the casino, you are assured the roulette wheel is fair (there is an equal chance of red and black and a small chance of green). You observe the first spin lands on red. Then you see the second spin land on red. Then the third. You keep standing there, and you keep observing spins land on red. How many spins do you need to observe before you become more sure than not the wheel is NOT fair? How would your bets change over time? How do your answers depend on the reputation of the casino? on who told you the roulette wheel was fair?

If you were the owner of the casino, at what point would you close the roulette table and have the mechanism of the wheel inspected?

2) You are assured by a friend a coin is fair. Your friend tosses heads. Then heads again. And again, and again. How may heads in a row do you need to see before you are more sure than not that the coin is not fair or your friend is not tossing the coin "randomly"? How does this depend on how much you trust your friend?

(A quick anecdote: In high school AP Statistics, our teacher asked everyone in the class to flip a coin 10 times and record the flips. More than half the class "surprisingly" recorded all tails or all heads on their flips. Several students claimed we had defeated statistics. We redid the exercise the with requirement that we had to flip the coin high in the air and let it fall on the ground several feet away from us. The teacher monitored the class more closely. Everything came out normal that time.)

3) A hurricane of record strength devastates the coast. People who didn't have insurance now buy insurance. Should they? Why didn't they buy insurance before? Are they irrational? How does your answer depend on your beliefs about the effects of global warming (in particular, your beliefs that over time the average strength of hurricanes will increase)?

No comments:

Post a Comment