tag:blogger.com,1999:blog-38600807.post49174644726145074..comments2018-06-02T14:19:34.554-04:00Comments on Advanced Football Analytics (formerly Advanced NFL Stats): Analytics and the Ludic FallacyUnknownnoreply@blogger.comBlogger54125tag:blogger.com,1999:blog-38600807.post-84256088881704145902012-11-08T16:54:27.543-05:002012-11-08T16:54:27.543-05:00Thanks Tarr. At this point, if some random intern...Thanks Tarr. At this point, if some random internet commenter claimed that Nate Silver breathed air and walked on two feet, I would ask for a citation.Mikenoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-12960623269815745652012-11-08T16:34:59.582-05:002012-11-08T16:34:59.582-05:00Mike,
Anon is wrong, and I have no idea where he...Mike, <br /><br />Anon is wrong, and I have no idea where he got that from. Silver gave McCain a 70% chance of winning Indiana in 2008.<br /><br />http://web.archive.org/web/20081219154455/http://www.fivethirtyeight.com/2008/11/todays-polls-and-final-election.html<br /><br />Also, anon, you seem to be thinking that "overestimate the uncertainty" means the opposite of what it means.<br /><br />Anyway, others have made the point, as I did, that these are highly correlated probabilities. One way to think of this is that the error of a given state's margin is a sum of the errors in the national popular vote margin, the regional margin, and the state margin.Tarrhttps://www.blogger.com/profile/14368810359650066790noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-9769138803987910212012-11-08T13:46:49.438-05:002012-11-08T13:46:49.438-05:00"in light of predicting indiana in 2008 as 10..."in light of predicting indiana in 2008 as 100% certain, and being wrong."<br /><br />Do you have a source for this? I recall he missed the vote percent by only a point. I would be surprised if that translated into 100% certainty.Mikenoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-70775814357104417272012-11-08T12:44:58.748-05:002012-11-08T12:44:58.748-05:00@ Marshall, thanks for pointing out the update. U...@ Marshall, thanks for pointing out the update. Unfortunately my RSS feed doesn't send me a 2nd link after an update or I would have gladly mentioned it myself. The stats are beyond me, I just wanted to share another perspective and that's why I included the link rather than make any argument myself. Glad to see it was updated.Boston Chrisnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-20570207710456931832012-11-08T12:31:28.681-05:002012-11-08T12:31:28.681-05:00Boston Chris - read the update to that article. As...Boston Chris - read the update to that article. Aside from ignoring the correlation between the states the author initially forgot the moe is two tailed and thus understated it by a factor of 2. In the update you find that 48 of 50 states were within Silver's moe projections (and assuming a 95% confidence interval you wouldn't expect him to get all 50) so he did very well.<br /><br />Anon - you would be correct if and only if the states were independent and not, as Silver predicts, highly correlated. The simple way to think of it is that it's less that there are, say, 6 states that are 60/40 Obama as that reality tends to be that it's 60/40 that Obama wins those 6 states. The coin flip analogy simply doesn't work - each coin flip is an independent result - election results are not and neither the model nor common sense looks at them that way. What you have to look at is not whether they went 60/40 in one way or the other but whether the results were within his predicted moe (which they were in 48 of 50 states).Marshallnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-62836536333419048912012-11-08T10:01:39.741-05:002012-11-08T10:01:39.741-05:00Tarr,
it is reasonable to guess that he did indeed...Tarr,<br />it is reasonable to guess that he did indeed overestimate the uncertaintly, in light of predicting indiana in 2008 as 100% certain, and being wrong. <br /><br />But the point remains: if he predicts obama to win at 60%, and romney at 40%, then romney has to win 40% of those. He won 0% of those. The probabilities were wrong. <br /><br />I haven't crunched the numbers to see how wrong it is, but this is one of those rare cases where you can take probability estimates, and determine if they were consistent with results, because of his amazing "success". <br /><br />To summarize with an analogy: he modeled his coin as 60% heads, 40% tails, flipped it 100 times, and it was heads 99 times. (note: made up numbers simply to illustrate the point, i'll have to look at the details).Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-9341047489722340182012-11-08T08:47:28.056-05:002012-11-08T08:47:28.056-05:00Harvard Sports Analysis did a review of how Silver...Harvard Sports Analysis did a review of how Silver did with his predictions, and not nearly the same conclusion on some here seem to be reaching. <br /><br />http://harvardsportsanalysis.wordpress.com/2012/11/08/nate-silver-and-forecasting-as-an-imperfect-science/Boston Chrisnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-70124269803932079142012-11-07T22:53:18.321-05:002012-11-07T22:53:18.321-05:00Gotta give credit where it's due. And it is du...Gotta give credit where it's due. And it is due.<br /><br />With regards to the request to revisit. I think the article stands as written: "This is not a criticism of how Nate's model works or what 538 does. On the contrary, I applaud and admire it. I have no reason to doubt Nate Silver plays things straight with his numbers. Nor do I have reason to doubt that, given the assumptions inherent in the model and the assumptions of the poll inputs, the probabilities of the 538 model are accurate."Brian Burkehttps://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-74342347808977819122012-11-07T20:11:16.551-05:002012-11-07T20:11:16.551-05:00Anon,
The short answer as to why is that those pr...Anon,<br /><br />The short answer as to why is that those probabilities are actually highly correlated.<br /><br />That said, it is likely that he overestimated the uncertainty.Tarrhttps://www.blogger.com/profile/14368810359650066790noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-16047578305914317712012-11-07T17:28:43.609-05:002012-11-07T17:28:43.609-05:00"Has anyone recognized the glaring (and I do ..."Has anyone recognized the glaring (and I do mean glaring) flaw in the Nate Silver analysis?"<br /><br />Ah yes, i see it has been mentioned. And indeed, looking at the 50 states, for 2012 and 2008, we see that Silver is 99 for 100. While many of his probability estimates are 100% to 0, there are several that are in the range of 60 - 90%. It is unlikely that all of those would be "correct". <br /><br />Additionally, one can calculate how unlikely it is, to get 99 out of 100 probabilities correct. When I get a chance, I will post it.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-35287600161175115612012-11-07T16:22:13.100-05:002012-11-07T16:22:13.100-05:00Has anyone recognized the glaring (and I do mean g...Has anyone recognized the glaring (and I do mean glaring) flaw in the Nate Silver analysis?<br /><br />Brian, is there a better way to communicate with you rather than a message getting lost in this comment list as an anonymous poster?<br /><br />It might make for a good article, and could possibly go national.<br /><br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-873842636815987392012-11-07T13:07:48.158-05:002012-11-07T13:07:48.158-05:00@Tarr: If you think I'm calling Silver a geniu...@Tarr: If you think I'm calling Silver a genius, you need to read all the comments before replying, lol.bigmouthhttps://www.blogger.com/profile/00336001437381184261noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-2492223929661017242012-11-07T12:13:20.235-05:002012-11-07T12:13:20.235-05:00If you call Nate Silver a genius, you really need ...If you call Nate Silver a genius, you really need to call Simon Jackman, Josh Putnam, Sam Wang, Drew Linzer, and probably others geniuses as well. There were a number of people looking at the polls, creating estimates and error bars around each state's result, and running monte carlo simulations to produce a national estimate. They had varying degrees of complexity (whether that's good or not) and some used some other inputs as well, but they ALL ended up with the same map, and that map turned out to be correct.Tarrhttps://www.blogger.com/profile/14368810359650066790noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-65697210991874754262012-11-07T11:07:09.352-05:002012-11-07T11:07:09.352-05:00So, it's the genius narrative! What's the ...So, it's the genius narrative! What's the over-under on when Michael Lewis writes a book about Silver?bigmouthhttps://www.blogger.com/profile/00336001437381184261noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-9491692983437210862012-11-07T10:54:32.026-05:002012-11-07T10:54:32.026-05:00Brian, how about revisiting this topic with a new ...Brian, how about revisiting this topic with a new look this time--given Silver's outstanding election swing states predictions?<br /><br />T.Dennis Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-1461444465527994352012-11-07T07:24:51.616-05:002012-11-07T07:24:51.616-05:00All great points, yet in the end: SILVER WAS GOLD!...All great points, yet in the end: SILVER WAS GOLD!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-23731052833692818682012-11-07T06:42:51.410-05:002012-11-07T06:42:51.410-05:00Looks like the non-ludic terms amount to a percent...Looks like the non-ludic terms amount to a percent or less.Andrew Folandhttp://nuclearmangos.blogspot.comnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-49473072756160524872012-11-07T01:20:49.049-05:002012-11-07T01:20:49.049-05:00Improbably, as of 1:15 on the east coast, it looks...Improbably, as of 1:15 on the east coast, it looks like Nate Silver is going to hit 100% this year. Although, to be fair, his final prediction was that Obama was only a 50.3% favorite in Florida, so for all practical purposes he didn't make a prediction there.<br /><br />Still, overall, the story here (from a stat-nerd prediction game perspective, of course) is that the polls were actually pretty darn accurate this year.Tarrhttps://www.blogger.com/profile/14368810359650066790noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-55919772229220379712012-11-06T16:42:24.879-05:002012-11-06T16:42:24.879-05:00what I have learned reading the post and all the c...what I have learned reading the post and all the comments:<br /><br />1) There are some very smart people who care about football scores,<br />2) The level of knowledge of statistics in these comments is intimidating to someone uneducated in such things but loves the logic of it all.<br />3) I'm fairly certain having finished reading all this I am far less certain of what I thought I might know. <br /><br />For the fun of it:<br /><br />Obama 332...my prediction. My sense is Silver's more right than Brian Burke thinks he is. I lean left politically and I am an optimist more than a pessimist (until Christian Ponder has to make a third down pass - then I'm not so optimistic).<br /><br />jmaronAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-33544473520610248312012-11-06T11:39:58.888-05:002012-11-06T11:39:58.888-05:00I'll say this: when all is said and done, Nate...I'll say this: when all is said and done, Nate's either gonna be celebrated as an infallible genius, or ridiculed as a total dunce, and neither narrative will really be correct, lol.bigmouthhttps://www.blogger.com/profile/00336001437381184261noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-86233217527405468232012-11-06T02:07:46.163-05:002012-11-06T02:07:46.163-05:00After rereading your post, I'm pretty sure I g...After rereading your post, I'm pretty sure I got it wrong the first time. I was thinking more in terms of what might be more properly called parameters (time left, rules) while you were saying that the polls held uncertainty that the score of a game does not right?<br /><br />In that case, you're certainly correct. But there is still a lot of uncertainty in both.<br /><br />Possibly more importantly the whole point of Bayesian analytics is to quantify and deal with that very uncertainty. Silver is trying quantify polling uncertainty and combine that with the other things that we know about elections and come up with a number that incorporates all that we understand about the election - not just polling but the history of a state, the effects of the economy on elections etc...<br /><br />Again, I take your point about the probability not being falsifiable and it's certainly a legitimate one. But I would still argue that we gain a lot from that sort of analysis - often far more than we would from either a naive reading of the polls or, God help us, just listening to the pundits pontificate (which somehow always ends up with why their guys is actually winning).<br /><br />Here's a question for you - is your objection primarily that he attaches a specific number to it? "Polls show Obama almost certain to win" doesn't seem like it would garner nearly the objections that "Obama has a 86% chance of winning" does.Marshallnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-12536083864750792422012-11-06T01:40:43.324-05:002012-11-06T01:40:43.324-05:00Brian - the idea that the sports model has certain...Brian - the idea that the sports model has certain inputs and the 'real world' has uncertain inputs is a bit arbitrary. Yes the rules of a game are fixed - but so too are the rules of the election (when the votes are cast, how they're counted, how electoral votes are divvied up). But, both are also loaded with uncertainty.<br /><br />Now you can certainly argue that there is more uncertainty in an election - and you'd be undoubtedly be right. But that's a difference in scale, not kind. Both a football prediction model and Silver's model have some fixed and many uncertain inputs.<br /><br />I take your point about the inability to falsify a probabilistic prediction. But what else would you suggest? You seem to be heading towards a perfect is the enemy of the good idea. Is polling perfect? By no means, but that doesn't mean it doesn't tell us something.<br /><br />Nate Silver would be the first to tell you the limits of what he's doing. Indeed he spells it out in his blog and his book. At the same time, I think there is quite a bit of value in it. <br /><br />I think many (not meaning you) fundamentally misunderstand the type of bayesian analysis he's doing and thus don't really understand his numbers - thus arguments about how many states or races he 'got right.' People seem to think that if he called 7 states 57% one way and they all went the way he called it would be a good result for him (which of course it wouldn't - he should only get the direction right on 4 of those).<br /><br />At this point his model is essentially predicting roughly a 10% chance that the polls are wrong. Is that number right? I don't have the data to argue one way or the other though I'm sure he does. Is it difficult to prove? Very much so - the best he can do is show how previous polling has done.Marshallnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-81130461766069842032012-11-06T01:19:29.405-05:002012-11-06T01:19:29.405-05:00Just as completion percentage and passing yards ar...Just as completion percentage and passing yards are a noisy measure of a QB's true abilities, so is polling data a noisy indicator of the electorate's true preferences. In both models, there is uncertainty in the thing you are truly trying to measure.<br /><br />"For all the attention Silver gets over his model from 2008, people seem all to eager to forget how wrong he was in 2010 when he completely missed the Republican wave which delivered the house."<br /><br />This has been repeated by a couple people, so I can only assume that it's a common talking point somewhere on the interwebs.<br /><br />It has a nice ring, but again, it's not actually true. The very first forecasts he released, in early September of 2010, predicted a Republican takeover.<br /><br />"He also incorrectly called 67% of the Senate races that year."<br /><br />Wow. This is not only wrong, it's hilariously wrong. There were 33 races, and he correctly predicted 30 of them - that's 91%, not 33%.<br /><br />Now, given that many races are very easy to call, missing 3 is actually pretty bad - there were maybe 9 or 10 races that were in any serious doubt in the closing months of the campaign, so his hit rate was much lower than on Senate races in 2008.<br /><br />It might bear noting that all three misses were in favor of Republicans. And the misses Silver had in the 2008 elections, small though they were, took the form of underestimating the margins for the Democrats. He did underestimate the Republican swing in 2010 (although he was close), but overall there sure doesn't seem to be any left lean in his predictions.Tarrhttps://www.blogger.com/profile/14368810359650066790noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-32845013591731342852012-11-05T23:48:06.705-05:002012-11-05T23:48:06.705-05:00538's swing state probabilities seem too extre...538's swing state probabilities seem too extreme (far from 50). <br /><br />The blog puts 4 swing states at above 80% likelihood for Obama:<br /><br />Virginia (80%), New Hampshire (85%), Iowa (85%), and Colorado (80%).<br /><br />Yet most of the poll data I've seen has the two candidates within the margin of error for those states.<br /><br />When I run a monte carlo simulation using more even odds that all favor Obama (taken from the Intrade state odds), I still get an Obama victory 70-75% of the time, but the electoral college MOV is nowhere near what Nate Silver is predicting.<br /><br />I hope he's right, or his supporters are going to get killed on the electoral college voting on intrade. <br /><br />Larry Sabato predicts are more moderate 290-248 victory for Obama. That prediction matches the mode from my simulations.Ianhttps://www.blogger.com/profile/07024027510665968270noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-52222149318839385422012-11-05T23:30:39.980-05:002012-11-05T23:30:39.980-05:00This just sounds too innumerate to be real.
We ar...This just sounds too innumerate to be real.<br /><br />We are only a week from saving hundreds of lives by making predictions in an open, dynamic, real-world system that has very little to do with games. <br /><br />I'd guess a plausible model is proven false when it is determined to no longer be plausible, but then it doesn't apply anymore. Neat.<br /><br />If Silver, Wang, Linzer, and everyone else are wrong about calling the overall winner (in the electoral college), it won't be their fault, as much as a systematic failure in polling in swing states which would basically drive every polling company out of business (yes, we could thank Obama for that). That is distinctly possible, but is it a 1%, 15%, 50% possibility.<br /><br />With the overwhelming number of polls in Ohio and other swing states, the statistical models being used by all these secondary people (Silver, Wang) don't even matter when picking the overall winner. The conclusion is plain as day.<br /><br />The overall winner is the only presidential prediction that really matters, but that isn't the only prediction being made. Some models will fare better with picking states (or congressional districts), margins, county by county results, or any of a number of different things. Even among those models that correctly pick the winner, there will be winners and losers.<br /><br />Just because there are only two possible outcomes doesn't mean that the outcomes are 50/50. The possibility of a black swan should not completely invalidate all of the other factors (like people telling us who they are voting for) that informed the prediction.Anonymousnoreply@blogger.com