Monday, December 30, 2013

Uncertainty (TFS, Part 11)

I am reading Thinking, Fast and Slow, by Daniel Kahneman. In this series I will summarize key parts of the book and supply some comments and reflections on the material.

Part IV: Choices
Chapters 25-34


Expected Utility Theory, Prospect Theory (gains/losses matter more than wealth; there is diminishing sensitivity to changes from the reference point; loss aversion; extremely low probability events are overweighted and extremely high probability events are underweighted relative to expected utility theory), The Endowment Effect.

If people have NO experience with something, then low-probability events are UNDER-weighted, not overweighted (e.g. perceived probability of an earthquake in CA by those living in CA who have not yet experienced an earthquake is too low).

Valuations of gambles are less sensitive to probability changes when the outcomes are vividly described. Framing matters. People can exhibit preference reversals, which violates standard economic assumptions.

Rules (Kahneman calls them "risk policies" in this part of the book), even ones that you impose on yourself, can mitigate some of these biases.

My Thoughts:

We have now reached the point in the book where things are becoming very dense. Much more dense than a normal "popular economics" book. If you are interested in reading more about the results and material in the book, I encourage you to pick it up.

Rather than spell out some of my "deeper" thoughts on this section, I want to present some of the paradoxes he uses in his (and others use in related) research. If you've never thought about the questions at the heart of these research paradoxes before (especially #3-5), they are worth thinking about.

Food for Thought:

1) "As a rule, I never buy extended warranties."

2) "Always take the highest possible deductible when purchasing insurance."


For the following questions, read them carefully, but decide on a preliminary answer in 10 seconds or less. Then, spend as much time as you need on them to reason out what you would do if it really mattered. (Hint: What would an expected value maximizer do? What would a utility maximizer who is risk averse do? Then ask again, what would you do?)

3) Decision i) Choose between:
    A) sure gain of $240
    B) 25% chance to gain $1,000 and 75% chance to gain nothing.

    Decision ii) Choose between:
    C) sure loss of $750
    D) 75% chance to lose $1,000 and 25% chance to lose nothing.

    Decision iii) Choose between:
    AD) 25% chance to win $240 and 75% chance to lose $760.
    BC) 25% chance to win $250 and 75% chance to lose $750.

If you chose A, D, then BC, think VERY HARD about what you just did.

4) Imagine the US is preparing for the outbreak of an unusual Asian disease, which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Assume that the exact scientific estimates of the consequences of the programs are as follows:
    If Program A is adopted, 200 people will be saved.
    If Program B is adopted, there is a one-third probability that 600 people will be saved and a two-thirds probability that no people will be saved.

Do you choose A or B? Now suppose instead your options were as follows:

    If Program C is adopted, 400 people will die.
    If Program D is adopted, there is a one-third probability that no one will die and a two-thirds probability that 600 people will die.

Do you choose C or D?

Think VERY HARD if you choose A and D. Think VERY HARD if you choose B and C. 

5) Suppose for a family, a standard tax exemption is allowed for each child the family has. The amount of the exemption does not depend on the income of the family. Should the per child tax exemption be larger for the rich than for the poor?

Suppose instead for a family, a tax surcharge is levied for each child fewer than three the family has. The amount of the surcharge does not depend on the income of the family. Should the childless poor pay as large a surcharge as the childless rich?

If you answered NO to both of these questions, THINK VERY HARD about your answers. You cannot logically reject both proposals. Was your reaction based on the moral framing of the questions or the substance of the policy?

6) Read about the Allais Paradox. Now that you know about it, would you make the same selection if offered the same gamble again? What if you could take the gamble 100 times and real money was on the line? What would you choose? Now that you've learned something, would any of your answers to 3), 4), or 5) change?

Thursday, December 26, 2013

Bring Back the BCS Computers! (TFS, part 10)

I am reading Thinking, Fast and Slow, by Daniel Kahneman. In this series I will summarize key parts of the book and supply some comments and reflections on the material.

Part III: Overconfidence
Chapters 21-24


Intuition often leads us astray. Simple formulas and simple statistics often out-predict experts in noisy environments. People are often hostile to algorithms.

Example. Trained counselors were asked to predict the grades of college freshman at the end of the school year. The counselors got to interview the students for 45 minutes and had access to high school grades, several aptitude test scores, and a four-page personal statement from each student. A simple regression based on only high school grades and one aptitude test did a better job of predicting the actual outcomes.

Planning Fallacy: Plans and forecasts are unrealistically close to the best-case scenario and could be improved by consulting the statistics of similar cases. (There is overoptimism on the part of planners and decision makers.)

How to mitigate the planning fallacy: Identify an appropriate reference class and obtain the statistics of the reference class. Use the statistics (not your intuition) as the baseline prediction.

End of Part III.

My Thoughts

Chapter 21 is, hands-down, the very best chapter in the book so far. In it, Kahneman explains the importance of using checklists and rubrics. He even has an interesting explanation of why checklists are so successful: they are like simple formulas; regressions without the weighting. So I need to take back all my griping about Kahneman not emphasizing checklists and rubrics from my previous posts! He does all the talking here!

On the BCS:

Every year college football decides a national champion. Next year, instead of the champion being determined by the outcome of the game between the #1 and #2 ranked schools (as determined by a combination of computer algorithms and coach's polls -- the BCS), the champion will be determined by a four team, single-elimination tournament. The four teams that play will be ENTIRELY determined by an important people/coach's poll/committee.

The goal of the change is to make the selection of the national champion less controversial. Instead, I think this move will make it more likely that the national champion will be controversial. Instead of choosing two teams, four have to be chosen, and now there is no defined criteria other than what the coaches feel like. I think the move to relying on the committee makes the decision process LESS structured and consistent from year to year.

One of the biggest perceived problems with the BCS people had was that a "computer" was choosing which one-loss team plays in the national championship. (In many years there is only one undefeated team, so the BCS had to select which other team to play against the undefeated team.) The move to a 4 team playoff chosen by coaches doesn't really solve the problem of which one-loss team(s) to include, because there are almost always more than four undefeated or one-loss teams. Instead, the decision will be less systematic and more controversial.

To me, the biggest problem with the BCS isn't that "computers" determine who plays but that the computer ranking algorithms were somewhat secret. Computers are great calculators. Things like strength of schedule and how to weight wins early in the season versus late in the season are debatable. Much better than letting coach's decide based on "feel" would be to publicly release the calculations done in the computer section of the BCS and be transparent about what they are trying to achieve. How is strength of schedule calculated? How much does an early loss matter? How much does a late loss matter? If there is a problem with the BCS computer formula, you should be able to identify WHY that problem exists, and adjust the formula from year-to-year.

At the very least, the algorithms should still be calculated and released even if only to provide information to the coaches. But based on the results Kahneman presents, there is no guarantee coaches will take these "computers" seriously, and will go out of there way to find "broken legs" the computer missed to overrule the ranking -- leading to poorer outcomes.

If I had to pick the ideal system for determining the champion (and who plays in the big bowl games), I would go with a Swiss-style tournament. The schedule throughout the season would be dynamic depending on the wins and losses of the previous week; the last round of the Swiss would be the championship game between the top two teams and the bowls. The season as a whole matters. A pure end-of-year tournament with lots of entries (like the NCAA basketball tournament) would make the end of the year matter almost exclusively. Teams only have to play well enough to make it into the tournament. There would be more games where the outcome doesn't matter. Even the worst team throughout the season of all the teams in the tournament could still become the national champion by playing the best in the tournament. To me, "national champion" should take into account the accomplishments of the whole year. Many people like the decisiveness of a tournament and the excitement of upsets, but just because you have a winner from a single-elimination tournament doesn't mean you've picked the best team of the year -- the national champion.

Friday, December 20, 2013

Luck or Skill? (TFS, part 9)

I am reading Thinking, Fast and Slow, by Daniel Kahneman. In this series I will summarize key parts of the book and supply some comments and reflections on the material.

Part III: Overconfidence
Chapters 19-20


Halo effect, hindsight bias, outcome bias, illusion of validity, illusion of skill (thinking an outcome is a result of mostly skill when the outcome is a result of mostly luck).

Errors of prediction are inevitable because the world is unpredictable.  High subjective confidence is not to be trusted as an indicator of accuracy (someone admitting low confidence could be much more informative).

My Thoughts:

One thing Kahneman complains quite a bit about is how much some people get paid when it seems like their job is mostly -- he stops just short of explicitly saying all -- luck. He calls out stock traders and CEOs specifically, and questions why incentive pay exists at all if it's almost all luck.

I think Kahneman goes too far. For example, I don't think Microsoft's stock price wouldn't have jumped so much at the announcement of the retirement of a lackluster CEO if CEOs don't matter.

In one calculation, Kahneman looks at investment results for 25 investment advisers in one firm over 8 years and finds basically 0 average correlation in the relative rank of the advisers across every pair of years (I am not sure looking at every pair of years and averaging to get an overall correlation is a good idea, but whatever). He concludes from the zero correlation that their job is "mostly chance" and the firm was rewarding employees based on "luck rather than skill," so why pay bonuses? It's very easy to imagine that without the bonuses advisers would have worked less hard for the firm and all achieved worse results on their investments or brought in fewer clients, or whatever, to the detriment of the firm. The bonuses serve a very useful purpose of encouraging harder work. The zero correlation in rank of the employees over time does not necessarily mean the bonuses were serving no purpose as Kahneman seems to think.

Food for Thought:

1) "You should have known the market was going to crash!"

2)  "We had access to all the information we needed. Why didn't the government connect the dots to prevent 9/11?"

3) Why do people trade stocks at all? (No Trade Theorem)

4) An engineering firm hires a new class of employees with exactly the same credentials from a good college. The firm cares about producing patents. The ability of employees to come up with patentable ideas is a function of their credentials, hard work/effort, and a lot of luck. The firm cannot observe effort or luck. Employees decide how much effort to put into their job.
    a) Will the firm produce more patents if it pays bonuses based on the number of patents rather than a flat wage?
    b) Will the distribution of income among employees with identical credentials be more unequal if bonuses are paid out?
    c) Alice received a $10,000 bonus this year. Bob received no bonus. True or False: We know for sure that Alice worked harder than Bob this year.
    d) Suppose all employees choose to exert the same amount of effort under the incentive pay system. The difference in patents (and pay) among employees at the end of the year is thus entirely due to luck. Does that mean the incentive pay system is serving no useful purpose? What if it was known that income will be redistributed by the government at the end of the year to make everyone's pay equal. Pay differences are due to luck, after all. Would employees produce as many patents?
    d) Suppose more firms move to incentive pay rather than flat wages causing inequality in the whole country to increase. Is this a good thing? Is it fair?

Saturday, December 7, 2013

Tom and Linda (TFS, part 8)

I am reading Thinking, Fast and Slow, by Daniel Kahneman. In this series I will summarize key parts of the book and supply some comments and reflections on the material.

Part II: Heuristics and Biases
Chapters 14-16


People tend to rely too heavily on representativeness and stereotypes when judging likelihoods. People are not good at intuiting Bayes's Rule or other probabilistic laws (especially when there is a good story to tell otherwise).

My Thoughts:

This section of the book is the weakest so far in terms of the quality of the research, robustness of the results, and the importance of the results in a broader context. I was going to rip it apart paragraph by paragraph, but Kahneman does a very surprising thing in chapter 15: he admits that it is the weakest body of research his ideas have sparked. I find this incredibly honorable (even though I still think the research is even weaker than he admits). One doesn't often see this in popular writings.

Case in point: I read Why Nations Fail last year, and while the book is good and from two great economists, the whole thing can be summarized in one sentence: Nations fail because of bad, extractive institutions while culture and geography don't matter. But the authors go way overboard in stating their case, reliance on questionable historical examples, and bashing opposing theories. The world is more complicated than they admit, and solutions are not as simple as clamoring for "inclusive institutions" (what does that even mean, anyway?).

But, I digress. I want to talk about two main studies, in this section of the book.

1) Profile of Tom W: "Tom W is of high intelligence, although lacking in true creativity. He has a need for order and clarity, and for neat and tidy systems in which every detail finds its appropriate place. His writing is rather dull and mechanical, occasionally enlivened by somewhat corny puns and flashes of imagination of the sci-fi type. He has a strong drive for competence. He seems to have little feel and little sympathy for other people, and does not enjoy interacting with others. Self-centered, he nonetheless has a deep moral sense."

Given the psychological profile of Tom W, people tend to be overly confident that he is a computer scientist. The error is that while Tom W may be close to the stereotypical/representative computer scientist, there aren't that many computer scientists relative to the population as a whole. There are many more people in humanities and education than computer science, so there should still be a significant chance (maybe even more likely) that Tom is in the social sciences. People seem to disregard the population base rate when given specific information that tells a good story.

However, as this study (which I've linked to before) summarized in the literature review/introduction, when researchers ask for frequencies ("how many out of 100?") rather than probabilities, base rate neglect disappears and subjects act like good Bayesians even when the subjects are not trained in statistics.

Here one more problem: Kahneman pulls a bit of a fast one in his description of the Tom W results. Rather than providing actual frequencies of nerdiness in different fields (which is important because stereotypes are real reflections of the environment and real selection effects and can be self-reinforcing), he claims people should have stayed close to the base rates because "the source of Tom W's description was not highly trustworthy." Say what? I mean, we know that and Kaheman knows that (he made up the description especially to fool a colleague -- it's not something that comes from an actual psychological profiling), but do the subjects know that? Probably not.

2) Linda. "Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations. Please rank order the following statements with respect to their probability:
    -- Linda is a bank teller.
    -- Linda is a bank teller and active in the feminist movement."

Of course, "Linda is a bank teller" is more likely. The probability rule is

                         Pr(A) = Pr(A and not B) + Pr(A and B).

In other words, the probability Linda is a bank teller is equal to the probability Linda is a bank teller and NOT active in the feminist movement plus the probability Linda is a bank teller and active in the feminist movement. Therefore, Linda is a bank teller must be more likely. The problem is that "Linda is a bank teller and active in the feminist movement" makes a good story. (The conjunction fallacy.)

However, if instead of "Please rank order the following statements with respect to their probability" we ask "To how many out of 100 people who are like Linda do the following statements apply?" subjects do not violate the probability rule. The results are very sensitive to how the question is asked.

General criticisms:

Probability is hard! Do people even know what that really means? Kahneman on the one hand says people substitute representativeness for probability because probability is hard but on the other hand discounts the research that asks for frequencies because people intuitively understand what probability means. I don't think he can have it both ways.

When there are no stakes, right answers and wrong answers don't matter. Thinking (especially about probabilities) is difficult and costly. One of Kaheman's students responded "so what?" when Kahneman pointed out he violated an elementary logical rule in the Linda study. Kahneman was discouraged by this. But he shouldn't be. Why should people think hard when they don't want to and there is no cost to getting it wrong?

One more thing. Kahneman mentions John List's studies on the market for baseball cards as confirming evidence for the existence of psychological biases in individuals. But (for example, this paper) even if individuals have strong biases and fall into all the traps psychologists lay for them, it may not matter at all at the market level. Markets punish mistakes and people learn. If you keep making mistakes, you will go bankrupt and be forced out of the market. The market participants may not have any idea what statistical rules they are learning, they just know they are doing better. They may even exhibit really bad biases in all other areas of life. But that market experience matters for that market. 

This is a critically important insight. We can make fun of economists all we want for modeling simple, perfectly rational individuals, when we know people are more complicated than that, but if at the market level these biases are competed out and the market looks like it's made up of simple, rational people, then that's all that matters.

Part II: Heuristics and Biases
Chapters 17-18


Regression to the mean.

My Thoughts:

These chapters are not that interesting. There is one really good quotation, though:

"Whether undetected or wrongly explained, the phenomenon of regression is strange to the human mind. So strange, indeed, that it was first identified and understood two hundred years after the theory of gravitation and differential calculus."

Amen to that.