- Jan 19, 2010
Crunchers, "Experts," and the Wrath of Randomness
Even if you don't believe people are perfectly rational, you might still expect decision-makers in sports—where there is an abundance of information, clear objectives, and severe consequences for failure—to get it "right." After all, these people are the "experts." There is no reason to think that some college professors armed with a slide rule can do any better.
Let's respond to that by noting that neither of us owns a slide rule (or knows how to use one). We do, though, have spreadsheets and some fairly sophisticated econometric software. There are a number of examples where people armed with such tools can see things that "the experts" miss. Some of our favorite examples come from places as diverse as the wine industry,13 analysis of Supreme Court decisions,14 and the treatment of heart patients in the emergency room.15 In essence, it appears that human beings—who are not actually lightning calculators—tend to lose in a contest against actual lightning calculators.16 Such an outcome is observed whether or not the human being is an "expert."
Related to the obvious point that people are not lightning calculators is a classic finding in psychology. People in sports often claim they can simply watch a player during a game and "know" if he is good or bad. The seminal work of George Miller, though, has shown that the human mind can only track about seven items at one time.17 In sports, though, a multitude of events are happening throughout the contest. All these events not only have to be seen and noted, the impact of these factors on wins must be ascertained. To claim that you can simply watch a player and see his or her overall contribution to wins suggests that you believe your mind can do something that research suggests is difficult. Despite the limitations of personal observation, though, human beings still tend to believe the analysis based on this approach is correct. Such overconfidence can often cause people to ignore contradictory information.
Statistical analysis, though, can overcome these issues. Spreadsheets and statistical software can evaluate more games than a person can ever personally observe. These evaluations can also allow us to look past the "most dramatic factors" and identify which factors truly matter most in terms of wins. Furthermore, the analysis can also easily change as new data arrives. Perhaps most importantly, statistical models come with confidence intervals.18 In other words, statistical models can assess the quality of the prediction being made. Try getting that kind of service from a human expert!
Number crunching does more than offer better explanations than what we get from "experts." It can also tell us when there really isn't an explanation. In other words, number crunching can help us see when a process is inherently random.
Let's illustrate this last point with an oddity from the Super Bowl. As of 2009, the National Football Conference (NFC) team has won the coin toss at the Super Bowl for 12 consecutive years. Such a streak clearly indicates that the NFC has some secret that allows it to better predict coin tosses; and the American Football Conference (AFC) better do some work if it hopes to close the "coin toss predicting gap." Then again, maybe there's another possibility. Flipping a coin is a random process.19 Even if you flipped a coin 12 times in a row with the same result, the process is still random. The outcomes don't tell us anything about the skill level of the NFC teams. This point should be obvious, since predicting a coin toss is not an actual skill.
This simple story highlights an additional advantage of analyzing sports data, and another potential pitfall for decision-makers. Some numbers that we associate with an athlete represent the skills of the performer. Other numbers, though, are not about a player's skill, but instead are determined by the actions of the player's teammates (or coaching or some random process). The analysis of numbers can actually clue us in on the skills versus non-skills argument. In the absence of such analysis, though, a decision-maker can actually suffer from the "wrath of randomness." Specifically, a decision-maker can be fooled by numbers that are as reliable predictors of the future as the numbers generated by our coin-flipping game. When that happens, money can be wasted on players who are not really helping. Or on the flip side, a player with some supposedly poor numbers can be removed from the roster when in fact the player is actually helping the team win.