I came across this article from a website called Trending Buffalo just yesterday. It's from September, but unfortunately it's one of the top 10 results if you google football analytics. Although the author is misinformed or uninformed or both, I sense that his feelings are shared by a large number of fans and traditional analysts in the media. So I thought I'd respond point by point.
From 'Trending Buffalo' Sep 13, 2013
1. YOU DON’T KNOW WHAT YOU DON’T KNOW
Precise breakdowns of every player on every play of every game can be found on countless websites [countless?] but “Player X failed to execute his route” and “Player Y is responsible for the B gap” ring pretty hollow when you don’t know what the players were supposed to do. [The author doesn't understand what analytics is. It's not amateur scouting.] Remember when Doug Flutie discredited analysts because “You watch the game. We watch the film”? He should’ve added “and we know the play calls.” Without that crucial information, your analysis is a really, really in-depth GUESS. [Football analytics is a lot of different things, combining tools from several disciplines, but it's not about guessing assignments on plays. I suppose the author doesn't care for Pro Football Focus, which isn't strictly 'analytics' on its own. But I'd submit that even though PFF doesn't know 100% of the assignments, most can be easily inferred. It's far from perfect, but it's a big step forward for public player evaluation and a giant leap beyond traditional punditry.]
2. WHAT HAPPENED 45 MINUTES AGO IS PROBABLY MORE RELEVANT THAN WHAT HAPPENED 45 YEARS AGO [What's the chance the author will get a green light at the next intersection? Is he 100% certain he will get the same result he got driving through it 45 minutes ago? Or is the best estimate based on a long-term average of recent history?]
You want to tell me I should’ve gone for it on 4th and 2 because of the success rate of the average offense versus the average defense on the average day since the merger? [I've never seen any analysis of 4th downs using data since the merger. The earliest data I've used is from 2000. However imperfect league baselines may be, they are still a massive improvement over a coach's intuition, and at the very least should calibrate a coach's risk tolerance.] Well, what if my offense is below average and my opponent’s defense is above average and the perennial Pro Bowl defensive tackle is eating my rookie interior linemen alive and my best available RB is nursing an ankle sprain? What do your numbers tell you now? [In that case, analytics can adjust the numbers for team strength, and further it can provide a coach a 'break-even' success rate, above which it makes sense to go for it.] Fact is– you’re not telling me what’s going to happen to me. You’re telling me what already happened to others. This is useful information and I’ll treat it as such… but that’s where it ends. [If someone told the author that 20% of the people who go swimming at that beach are attacked by sharks? Would he discount that information and swim there because it's something that "already happened to others?"]
3. NOW NERDS CAN PLAY! [Nice smear.]
Football is a big, tough manly sport and analytics provide[sic] a way for all of us to get involved without getting hit. It’s appealing. We value our ability to think our way through the game and, after the fact, it makes a lot more sense to say “I wouldn’t have done what that coach did” than “I wouldn’t have thrown that interception” or “I wouldn’t have fallen for that juke move.” [The author tries to define football analytics here as outcome bias with numbers. Knock that straw-man down with all your might!] We can imagine ourselves in the role of coach. We all have Madden. We all have Excel. Plugging flawed data (see #1) into a spreadsheet in an attempt to accurately predict results is a fool’s errand. [All analysis using flawed data would be a waste of time. What about when the data isn't so flawed?]
4. IT’S (MOSTLY) PSEUDO-SCIENCE
I’m 100% in support of professional teams utilizing all of the data at their disposal in an effort to make the best decisions possible [Ok, great. So we are in agreement] but it all breaks down when we get to the amateur/armchair level. [With some exceptions, the amateur/armchair level is two decades ahead of the pro teams in terms of successful applications of analytics.]
Pseudoscience is a claim, belief, or practice which is presented as scientific, but does not adhere to a valid scientific method, lacks supporting evidence or plausibility, cannot be reliably tested, or otherwise lacks scientific status. Pseudoscience is often characterized by the use of vague, contradictory, exaggerated or unprovable claims, an over-reliance on confirmation rather than rigorous attempts at refutation, a lack of openness to evaluation by other experts, and a general absence of systematic processes to rationally develop theories. A field, practice, or body of knowledge can reasonably be called pseudoscientific when it is presented as consistent with the norms of scientific research, but it demonstrably fails to meet these norms.[Football analytics is precisely the opposite of what the author quotes above as pseudoscience. Specifically, analytics seeks to take sports analysis out of the realm of the intuitive and the post-hoc narrative, and subject it to rigorous scientific methods. It's objective, precise, consistent, evidence-based, makes falsifiable predictions, and is open to evaluation. There are some kooks out there who claim to be doing analytics, but that's a fact of life in any field of inquiry.]
The lack of reliable data (primarily because of the inability to use a control group to isolate specific events within the context of an 11 on 11 game) severely injures the credibility of any conclusions derived from said information [Huh?]. So, the average NFL play nets 4.8 yards? How many yards is the average team attempting to gain on each play? 2? 4.8? 80? Without that knowledge, we have nothing. [Analytics is not about predicting the outcome of the very next play or mind reading. Where did the author get that impression?]
5. IT’S GETTING WORSE [It's gaining traction because it's useful. And it apparently worries the author because he doesn't understand it, as his article demonstrates.]
Looking at data before making decisions is a positive development and I hope every team I support keeps up with the times. [The author should just stop right there. That's all analytics claims to be. There are statistical and analytical techniques that make sense of that raw data, but those same techniques are applied successfully in all fields of industry, government, and science.] But the further we go down this road in popular/public/amateur conversation, the more bad information creeps in. Suddenly, the word “analytics” starts to sound a lot like “run and stop the run” and “defense wins championships.” Research all you’d like but we’re getting into “He made the wrong decision and I can prove it” territory (even though you can’t) and it’s growing tiresome. The day that “Analytics Guy” [I wonder who he's referring to?] realizes his information is helpful but not definitive, everybody wins. [Well, somebody is winning this debate, and it's not the author of this article.]
And besides, if you don't care for the kind of analysis here at ANS, we helpfully provide an alternative.