By Brian Burke
One of the most visible applications of analytics in this year’s election is the FiveThirtyEight blog, created by Nate Silver. For those who may not be familiar with Nate, he is a noted baseball sabermetrician best known for creating the PECOTA prediction model. His election forecast has favored Obama more heavily than most others throughout the election season, and he has been the target of criticism recently.
In defending his approach, Nate and others have explained his probabilistic reasoning with examples from football. The 90% or so chance Nate gives Obama to win the Electoral College is, for some reason, put in football terms…Romney is down by 3 with 2 minutes to play…or Romney is down 7 with 5 minutes to play…or something along those lines. (I think that's ironic given that even football experts don't seem have a good grasp of situational probabilities.)
I disagree with those analogies, but not because I have any better reason to think that either candidate will win. I think the situation is more like this: Romney is down by a very small number of points with 1 minute to play, and we don’t know who has the ball or where the line of scrimmage is. Or maybe it’s more like this: Romney is down by a point or two and has just snapped a long field goal attempt, and no one has a very good idea which way the wind is blowing. If it was blowing just like it was last game, the kick will almost certainly come up short. But if the wind is blowing more like it did two games ago, he’ll probably make the kick and win.
The distinction is in the level of uncertainty of the model inputs. In a football game, we know the score, time, possession, down, and distance with absolute precision. In contrast, one of the most critical inputs in an election as close as this year's will hinge on enthusiasm, which can only be estimated and relies heavily on uncertain assumptions.
Black Swan author Nassim Taleb coined the term Ludic Fallacy to describe the mistake of projecting the certainty of game analysis onto real world analysis. (Ludus is Latin for ‘game.') When we analyze sports and other games we can be relatively certain of our conclusions because they are bounded systems. For example, in football the field is always 100 yards long, there are always 4 downs, regulation time is always 60 minutes, and a touchdown always gives you 6 points plus an extra point (and sometimes 2). My win probability model is complex and flawed enough knowing that those rules are fixed constants. Imagine trying to make a win probability estimate not knowing how long the game might be, or how many downs each team will have, or how many points each score might be worth. Or imagine that any of those things might change at any moment.
The real world is much messier than any game. The real world is an open system of infinite complexities and interactions, and often subject to the fickle whims of human emotion. The confidence we have in understanding and predicting outcomes in games with their bounded constraints should not be extrapolated into the untidy realm of the real world. The confidence we gain from modeling relatively simple systems like a sports season can quickly become overconfidence when making, say, economic projections or predictions of sociological trends.
One of the deficiencies of any statistical/probabalistic model (including any of my own) is that they are unfalsifiable. According to philosopher Karl Popper, when something is impossible to falsify, it's not really science at all. If I say the Giants have a 65% chance of beating the Steelers but they lose (which they did), no one can say I was wrong. After all, I said the Steelers had a 35% chance. Only after large number of probability estimates are tested by true outcomes can we assess the accuracy of a model. And by then, the nature of the social process or system may have changed to such a degree to render the model obsolete. Sports and other games usually provide the number of tests to give us the sample sizes we need for confident evaluation, but do recessions...terror attacks...presidential elections?
There are some unfair attacks on Nate and his model to be sure, but the folks who are skeptical of predictions from analytic models of complex systems are not 'against science' or 'anti-math' as they are often portrayed. They have a sound epistemological basis for caution. I put myself in the ranks of skeptics who question the ability of economic and sociological models to make sound predictions with high levels of certainty.
That’s not to say we can’t learn anything from modeling real world systems or we shouldn’t try. Of course analytics can make useful information out of data. But because of the interaction of uncertainty and complexity, we are far better at explaining the past than predicting the future. Explanation is easier because there is far less uncertainty in the inputs, and the relationships between variables can be known. Just ask the mortgage-backed securities analysts on Wall Street about the dangers of over-certain projections in real world systems.
This is not a criticism of how Nate's model works or what 538 does. On the contrary, I applaud and admire it. I have no reason to doubt Nate Silver plays things straight with his numbers. Nor do I have reason to doubt that, given the assumptions inherent in the model and the assumptions of the poll inputs, the probabilities of the 538 model are accurate.
My only commentary is a warning about how certain we can be about projections of unbounded, real world systems. In an election this close, a swing of just a few percentage points of enthusiasm and turnout would change the projection from 90% Obama to 90% Romney. There seems to me to be too much leverage in something so uncertain to be confident of anything beyond the 60% level. This is strictly my personal philosophical stance, and I don't pretend to be an expert on polls.
Nate Silver has recently written a book about prediction, and I'd bet that he's well aware of the points I'm making and probably writes about them more eloquently than I could. I look forward to reading it. Perhaps the 538 prediction model accounts for the uncertainty I'm pointing to, but the top-line poll inputs the model relies on almost certainly do not. Unfortunately, the model is proprietary, so we can't know for sure.
I don't usually do this, but I'm going to go out on a limb and make a prediction in the presidential race. It's 50/50. Prove me wrong! The kick is in the air, and it all depends on the wind.