tag:blogger.com,1999:blog-38600807.post6191744191600089030..comments2016-07-21T09:07:00.859-04:00Comments on Advanced Football Analytics (formerly Advanced NFL Stats): Pre-Season Predictions Are Still WorthlessBrian Burkenoreply@blogger.comBlogger50125tag:blogger.com,1999:blog-38600807.post-37509296589284709732014-07-31T09:16:10.509-04:002014-07-31T09:16:10.509-04:00Good stuff, I just bought the FOA 2014 and was won...Good stuff, I just bought the FOA 2014 and was wondering about the relevance of the predictions. I started an excel file with the Mean projection and the "bracket" projection (on the clock, mediocrity ...) for year n, comparing it with n+1<br />Even w/o any sound calculation like MAE or RMSE, it's pretty obvious that those predictions don't hold up.<br />The only thing you can take out of the predictions is that if a "bracket" is 15% more than the next, there's a 80% chance that will happen.<br />In FOA 2011, CAR was seen as a 4-6 win team at a 45% probability, the next bracket being 7-8 at 11%. They ended up 6-10.<br /><br />Anyway, there's so much more interesting stuff in FOA for 10€ that I don't mind the predictive part being poor. Still, it's important to know it.Arthurohttp://www.blogger.com/profile/11113171382174269410noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-22912447988165705802010-09-21T17:27:55.271-04:002010-09-21T17:27:55.271-04:00The "experts" picks are as useless as a...The "experts" picks are as useless as as the teats on a boar hog.<br /><br />Injuries, suspensions, weather, and many other factors have to be considered.<br /><br />Vegas odds makers are far better suited to know <br />what will happen than retired "jocks" in neck ties<br />sitting behind desks pretending to know what will happen.<br /><br />Dallas 0 and 2 Ask those who pick them to be super bowl bound (it could happen, BUT).<br /><br />Tampa Bay 2 and 0, (they are hopeless) Yata, yata, yata. Watch out.<br /><br />There are surprises to come, but don't bet on it.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-21586367893701859202010-09-09T10:58:05.908-04:002010-09-09T10:58:05.908-04:00PS: Shouldn't you compare the predictions to t...PS: Shouldn't you compare the predictions to teams' Pythagorean winning percentage?bigmouthhttp://www.blogger.com/profile/00336001437381184261noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-29170079943737905472010-09-09T10:54:55.964-04:002010-09-09T10:54:55.964-04:00And here, in a nutshell, is why passively managed ...And here, in a nutshell, is why passively managed investments are better than actively managed ones.bigmouthhttp://www.blogger.com/profile/00336001437381184261noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-19999489816362059072010-08-06T17:18:47.315-04:002010-08-06T17:18:47.315-04:00barawn isn't fooling anyone. you don't use...barawn isn't fooling anyone. you don't use the prediction's standard deviation (sigma). you would use the actual distribution's standard deviation! but that's not the point in the first place. no one cares how many standard deviations a prediction is off. what matters is the absolute error.<br /><br />he either doesn't understand or is being deceitful.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-17127047867696311232010-08-06T16:25:40.125-04:002010-08-06T16:25:40.125-04:00Ya know.. If you close your eyes and walk across a...Ya know.. If you close your eyes and walk across a busy freeway in rush hour you have a pretty good shot at making it to the other side because traffic is likely to be at a complete stand still.. On the other hand...it might not be and your ears might be plugged up and you won't hear the whoosh of cars speeding by at 90 mph.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-9559636461426265332010-08-06T13:11:09.833-04:002010-08-06T13:11:09.833-04:00I am only snide to people who use the words "...I am only snide to people who use the words "clearly" and "obviously" in their comments. It betrays their agenda.<br /><br />No, it's not perfectly normal. Nothing really is in statistics. <br /><br />If you want to credit FO for their distribution prediction, then we'd have to give the credit must go to the naive predictions too. But that begs the question. What is the point of a prediction that is distributed so large it is meaningless?Brian Burkehttp://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-37938022027395249852010-08-06T13:05:32.663-04:002010-08-06T13:05:32.663-04:00Is there some reason you've just started becom...Is there some reason you've just started becoming snide in most of your responses? I'm not criticizing just for criticism's sake. Certain predictions from FO are very uncertain (any which involve coaching changes) and several are very certain (those which involve the Colts and Patriots, for instance).<br /><br />Lumping them all together - <em>when they tell you, flat out, what the uncertainty is</em> is just bad statistics. Stating that "it's so bad, it's like negative football information" is worse than that - it's a silly analogy that doesn't do anything but lower the bar of the conversation.<br /><br /><em>The distribution of NFL wins is reasonably Gaussian</em><br /><br />Not from your own research.<br /><br />http://www.advancednflstats.com/2007/08/luck-and-nfl-outcomes.html<br /><br /><em>and RMSE is a fair and well-suited way of evaluating the error.</em><br /><br />No, it's not. You're just looking at the error of the mean prediction. That's just ridiculously simplistic.<br /><br />Comparing a prediction by <em>just</em> its mean value when it has a distribution is <em>of course</em> going to make an "idiot prediction" (i.e. one static value close to the mean, or close to an existing distribution) look better when you've got a distribution this wide (<a href="http://eaglesbythenumbers.blogspot.com/2010/08/preseason-predictions-how-to-judge-them.html" rel="nofollow">see here</a>).<br /><br />FO gives a likelihood distribution for each prediction. How do you test that prediction? You lump together all teams which are predicted to win 8 games, say, 40-50% of the time. Then you compare it to how often they <em>do</em> win 8 games. Then you lump together all teams which are predicted to win 8 games 30-40% of the time, and compare it to how often they <em>do</em> win.<br /><br />Or, if you want to be really simple: compare the RMSE to the <em>expected</em> RMSE based on the width of the distributions given by FO. It's not a <em>good</em> test, because they're not Gaussian, but hey, if you insist they're close enough, sure.<br /><br />Koko the Monkey's predictions and the CoMA predictions will be <em>awful</em>, because they've got a ridiculously low standard error (0.3 games, the Gaussian equivalent width of a uniform distribution).<br /><br />If memory serves, the FO predictions have a standard error of about ~2 wins. Which means that the FO predictions have an RMS error of about ~1.5 sigma, whereas the CoMA/Monkey predictions have an RMS error of ~10 and ~9 sigma respectively.<br /><br />Fuzz out the CoMA/Monkey predictions by ~2 wins and their RMSE will be <em>larger</em> than FO's predictions, adding in quadrature.<br /><br />In any case, this isn't an incredibly complicated things. FO gives a distribution of expected wins, you've got a distribution of <em>resulting</em> wins, and you're only comparing the first moment of each distribution.barawnhttp://barawn.livejournal.com/noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-7980566569405481592010-08-05T14:49:46.728-04:002010-08-05T14:49:46.728-04:00barawn-Hey, you're right! I ran a K-S test and...barawn-Hey, you're right! I ran a K-S test and it turns out the Rams won 9 games and Washington won the NFC East. Who knew?!<br /><br />The distribution of NFL wins is reasonably Gaussian, and RMSE is a fair and well-suited way of evaluating the error.Brian Burkehttp://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-56127411780862390302010-08-03T16:14:47.190-04:002010-08-03T16:14:47.190-04:00Keep it up, Brian! Not everyone is missing the poi...Keep it up, Brian! Not everyone is missing the point of your article!Jameshttp://www.blogger.com/profile/01838293735141324662noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-74256633933610783232010-08-03T12:09:52.527-04:002010-08-03T12:09:52.527-04:00For the more technical-minded: the reason mean and...For the more technical-minded: the reason mean and RMS are used is because the mean is the first moment of a a Gaussian distribution, and the RMS is the second moment. So you're essentially doing a poor-man's K-S test, assuming the underlying distributions are Gaussian. The moments of the win predictions given by FO are clearly not well described by mean/RMS.barawnhttp://barawn.livejournal.com/noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-72615654007862254482010-08-03T12:05:03.735-04:002010-08-03T12:05:03.735-04:00This really isn't the right way to test predic...This really isn't the right way to test predictions.<br /><br />RMSE only means something statistical if the distribution's Gaussian. It's not. It's given in the Almanacs, and it's clearly non-Gaussian.<br /><br />If you have a flat distribution from 6-10 wins, you'll have a mean of 8 wins. If the team then wins 6, you'll say "ah ha, your error was 2 wins"... but the result was actually perfectly expected. You started off by saying "the team will win between 6 and 10 games, and they all seem perfectly likely" and in the end, that's what happened. Looking at mean error and RMSE is a terrible way of evaluating a prediction that gives a non-Gaussian distribution.<br /><br />You want to do something like a Kolmogorov-Smirnov test. We already know that a K-S test based on Koko the Monkey is crap - the win distribution for teams doesn't look like a random draw around 8 wins.<br /><br />Although it should be stated that if you tried to do a K-S test by drawing randomly from each team's given probability distribution also isn't quite right, because the probability distributions aren't independent and you'd be drawing them independently. You really want to compare the *overall* season results with the likelihood from the simulated season results. <br /><br />For that you'd have to get the simulated season results, which you could probably get by asking.barawnhttp://barawn.livejournal.com/noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-16384694822547272742010-08-03T00:23:06.306-04:002010-08-03T00:23:06.306-04:00It appears you are referring the FO "2009 Odd...It appears you are referring the FO "2009 Oddities" post linked in comment #1, where they showed that offensive year-to-year correlations went through the roof and defensive year-to-year correlations fell off the table. It has nothing to do with DVOA or a "magical black box", as they showed the same effect using points scored/allowed.<br /><br />Obviously the key to projecting future performance based on past results is knowing how much to regress toward the mean. Such a seismic shift in y-t-y numbers would throw any model off track regardless of what other bells and whistles are added in to the mix. This finding is fascinating to me and should be to anyone interested in performance projections or team building. Will the trend continue? Should GMs now allocate more of their resources on offense? It's truly unfortunate that any researcher would avoid the opportunity to explore the issue further out of some apparent vendetta against another researcher.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-66037375746077655782010-08-02T16:56:50.448-04:002010-08-02T16:56:50.448-04:00You've accused me of "blindly bashing FO,...You've accused me of "blindly bashing FO," but one thing we can agree on is we're all blind when it comes to their numbers, yourself included. It's a magical black box, impervious to criticism. It can only be judged on the results it produces, which is what I've provided here. If that's not interesting or useful to you, feel free to go back to reading your Ouija board.<br /><br />True, the RMSE for Vegas over those particular 4 seasons was 3.1 while FO's was 3.0 (about the same difference as a coma patient and FO). However, with 2009 added, it becomes a wash at best. Additionally, the Vegas Watch link does not account for the odds given for each team. For example, the over/under on the Ravens might be say 10, but to bet the under (9 or fewer wins) you have to pay 120 to win 100, meaning the real over under is not 10 but 9 or so. <br /><br />To answer your question, I wouldn't buy that explanation without seeing the data. Just looking at their list of predicted team win totals for 2009, it appears they predicted some teams to improve drastically, well outside what a normal year-to-year improvement would suggest. In other words, FO was not burned by relying on year-to-year correlations in performance, but because they deviated too far from them.Brian Burkehttp://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-84463397058589692172010-08-02T15:43:05.549-04:002010-08-02T15:43:05.549-04:00According to the Vegas Watch blog, FO projections ...According to the Vegas Watch blog, FO projections went 55-41 (.573) against the Vegas Over/Unders from 2006-2008. That seems pretty good to me.<br /><br />http://vegaswatch.net/2009/07/football-outsiders-vs-vegas-2007-2009.html<br /><br />2009 was an outlying season in several ways, as FO has readily admitted and documented. Their projection system regresses offense and defense separately and for some reason the coefficients measured from the last decade didn't fit last year at all. Instead of blindly bashing FO, I'd rather see an intelligent discussion about a) whether you've confirmed this with your own numbers, b) why you think this might be so, and c) what it might mean for the future of the NFL (if anything).<br /><br />I'm also curious why you're so sure that all preseason predictions are worthless, yet at the same time you're convinced that your own game prediction system can beat Vegas. Have you attempted to create your own season projection model and failed?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-39228980570502631502010-08-02T14:30:04.878-04:002010-08-02T14:30:04.878-04:00A few thoughts:
#1 This is not cherry-picking one...A few thoughts:<br /><br />#1 This is not cherry-picking one errant year. I've seen that comment here and on a few other sites which linked to this article. First of all, 2009 is the most recent year, so of course that's relevant and not cherry-picked. Besides, FO's track record is very poor going back five seasons, essentially tying the far simpler methods. See the link I posted above for previous years.<br /><br />#2 It's fine if predictions are intended for entertainment purposes only. But that's not the case here. Is the point that FO is nothing better than the football equivalent of the psychic hotline? <br /><br />#3 If you sell analysis and predictions, be prepared to be held accountable, especially if the analysis is a statistical black box.<br /><br />#4 I agree, and have been on record previously, that FO produces very fine non-statistical analysis. It's no worse than 99% of the superficial stuff we read everyday on espn.com or anywhere else.<br /><br />#5 However, if their statistical projections are informed by their qualitative analysis, or their qualitative analysis is informed by their stats, what does that say about the quality of their "other stuff?" Further, one big reason why people buy into predictions is the level of detail in the basis of the projections. People are far more likely to believe fake stories with rich detail.<br /><br />#6 I'm picking on FO because they claim to have a very technically advanced and complex prediction system and they sell their products based on their claims of technical analysis, not for any other reason.<br /><br />The point is, unless you have an open, documented, proven method for predictions, all you're doing is palm-reading. Let's face it, all the pre-season analysis, projections, and predictions are nothing better than tarot card/Ouija board/crystal ball/fortune cookie nonsense. But people still eat it up, despite knowing better. Let's just not pretend it's anything more than it is.<br /><br />Also, it's remarkable how some FO defenders react emotionally. I think there is some deeper stuff going on, namely cognitive dissonance. If a customer regularly forks over money for a product, which later turns out to be bunk, then the customer has a decision to make. He can either admit he's been a fool, or he has to somehow preserve the illusion that the product is worthwhile. No one likes to admit folly, even (or perhaps especially) to themselves, so people do all sorts of psychological contortions to preserve their self-image of competence.Brian Burkehttp://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-23354023325912690362010-08-02T13:33:45.724-04:002010-08-02T13:33:45.724-04:00If all FO did was make predictions, I think this w...If all FO did was make predictions, I think this would be more relevant. Instead, you are picking on a site that offers great insight into teams and then offers probabilities. <br /><br />Were the Jets really very good last year? They caught a massive break the last two weeks of the season or they well could have finished 7-9 and the FO prediction wouldn't have looked so bad.Nate Dunlevyhttp://18to88.comnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-87352171782512250892010-08-02T12:18:11.561-04:002010-08-02T12:18:11.561-04:00I'm not even a stats-geek second, much less a ...I'm not even a stats-geek second, much less a differently-gruntled former FO employee...but isn't this a sample-size problem? Wouldn't one always expect significant variance with any prediction, even if one ran a great simulation 10000 times?Jonathanhttp://www.blogger.com/profile/11189860457582272745noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-82295906194459922432010-08-02T11:55:29.803-04:002010-08-02T11:55:29.803-04:00Will Koko be comparing his fantasy rankings from 2...Will Koko be comparing his fantasy rankings from 2009 to the actuals? <br /><br />http://www.advancednflstats.com/2009/08/koko-fantasy-football-monkey.htmlAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-44991419475680990012010-08-02T11:39:19.311-04:002010-08-02T11:39:19.311-04:00what about Paul the Octupus? explain that one! :)...what about Paul the Octupus? explain that one! :).asking for a "friend"noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-60455889044867929552010-08-02T10:37:38.755-04:002010-08-02T10:37:38.755-04:00"I think Alchemist is really on to something...."I think Alchemist is really on to something. One of the most important things about predictions is that they are supposed to be "entertaining". They are designed to sell magazines and subscriptions right now not to win a five year statistical study five years in the future."<br /><br />Yes, but these predictions lose much of their entertainment (or persuasive) value when you already understand how meaningless and gimmicky such predictions are to begin with.Jonathannoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-53203145406384941872010-08-01T08:24:54.076-04:002010-08-01T08:24:54.076-04:00"Regression is engineered in the NFL, they sc..."Regression is engineered in the NFL, they schedule to ensure parity.. and then 2 teams accounted for 5 of the last 10 championships.. "<br /><br />This is a myth that has become repeated by so many lazy reporters that it is now taken by most people as fact. Out of 16 games, there are a grand total of TWO of them that have anything to do with the record of your opponents. <br /><br />Each team doubles up against division opponents - that gets us to 6 out of the 16 games. Then they play against all the teams from another division within their conference - that gets us to 10 of the 16. Then they play against all the teams from another division in the opposite conference - that leaves only 2 of the 16 games that can be scheduled based on the previous year's record. <br /><br />Now if I'm a division winner, those two "parity games" will be against other division winners, and I should, on average, split those games. This means that I will, on average, have one more loss than I would have expected if I had played two doormats. So on average, the "parity scheduling" hangs one extra loss on each of last year's division winners. <br /><br />Does that extra loss make a difference in the standings? In a short 16-game season, sure it does. But it's a far cry from saying that the NFL "schedules to ensure parity". For any team in the league, I can tell you right now who 14 of their opponents will be in 2020 - and it has absolutely nothing to do with the relative strength of the teams involved.<br /><br />Please don't take this to mean that I am downplaying the importance of scheduling in determining a team's win-loss record. One of the largest factors in a team's fortunes is whether it gets lucky or unlucky by being paired against relatively-weak or relatively-strong divisions within its conference and the other conference. For example, it would be really lucky for a team to be paired against all the teams from the NFC West and the AFC West, as these teams look to be relatively weak. But getting such a schedule would be based on luck - not on any parity scheduling.Alchemisthttp://www.blogger.com/profile/15989876051555196561noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-15604834403100978032010-07-31T23:12:38.262-04:002010-07-31T23:12:38.262-04:00so how do we do it better?
I would contend schedu...so how do we do it better?<br /><br />I would contend scheduling has more to do with the season a team has than who is on the roster, provided you have a pretty good roster.<br /><br />Regression is engineered in the NFL, they schedule to ensure parity.. and then 2 teams accounted for 5 of the last 10 championships.. <br /><br />Their engineering has made it so that there is huge turnover in teams that make the playoffs, 50% with no real change in who the ELITE teams are.. <br /><br />NYG, IND, PIT were in 2 super bowls in last ten years. NE was in 4. That's 4 teams accounting for 10 of the 20 super bowl spots. Is that parity?Jeffnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-71574330445847128372010-07-31T20:20:26.532-04:002010-07-31T20:20:26.532-04:00I think Alchemist is really on to something. One ...I think Alchemist is really on to something. One of the most important things about predictions is that they are supposed to be "entertaining". They are designed to sell magazines and subscriptions right now not to win a five year statistical study five years in the future. That means there is a definite need to go out on a limb. Making a prediction like Koko does that says "I predict everything the same but with regression to the mean thrown in" isn't going to sell anything.Jeff Clarkenoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-66337199132476724642010-07-31T11:18:51.960-04:002010-07-31T11:18:51.960-04:00lol love itlol love ittimnoreply@blogger.com