By Brian Burke
I think at the NFL level, all coaches employ the same best practices. There is no secret sauce that one coach has over another in terms of instruction, motivation, strategy, etc. This is because of the highly mobile, fluid market for coaches and the large size of their staffs. There are very strong constraints on deviation from league norms in any dimension.
Also, from statistical analysis, we can measure the variance in team performance attributable to randomness (sample error due to a short 16-game season) and player impacts (the addition or subtraction of a player's impact on team production, player interaction effects). There is very little variance left that can be attributed to other causes, including coaching. In other words NFL outcomes are overwhelmingly driven by player talent and luck, and there's not much room left for coaching to make a big impact.
We can observe this intuitively, as the very same coaches can have wildly different records from year to year. How much effect can they really have?
That said, if you made a clown like me head coach of the Ravens last year, I guarantee they would not have won the SB. So coaching obviously matters. It's just that NFL coaches are more or less equal (equally excellent), for the reasons stated above. In a competition system where participants are roughly equivalent, other factors will dominate the outcome. You can still really screw things up if you don't know what you're doing, but it's exceptionally difficult to stand out as superior.
Consider a simplified analogy. Rock Paper Scissors is a game where there is skill involved. The skill is: don't be predictable. Once everyone has that skill, outcomes are determined by other factors--in the case of RPS, it's randomness. It would be very easy to lose on purpose by acting unskilled but very hard to win on purpose, which suggests skill is involved. But if everyone is on the same skill level, skill becomes irrelevant.
I'm not saying that NFL coaches with poor records are losing on purpose. To be clear,I'm suggesting is that it's much easier for coaches to be on the downside of the equation than on the upside of the equation. There is an asymmetry to the top end of the "coaching function." In graphical form it might look like this:
If the x axis is coaching skill, then all NFL coaches, even the relatively "poor" ones, are going to come from the right side of the graph. Consider the y axis to be "effect of coaching on team outcomes." In the top echelons of coaching, there are decreasing returns to better and better coaching. As you move right along the curve, there is less and and less room for positive effect. But if you move leftward, the slope is steeper, which means it's easy to have a negative effect, but hard to have a positive effect as a coach.
(As an aside, this curve may explain why there is so little risk taking among NFL coaches. The perceived upside is smaller than the perceived downside. In most cases, this may be true.)
I realize this is not a widely accepted view. People attribute group success and failure to leaders for many reasons. First, it's in our nature. The fundamental attribution error causes people to put blame for failures on others but claim to be the cause of success. So the coaches who happen to lead winning teams are the first to perpetuate the myth of great coaching. The second group to perpetuate the myth are GMs and owners who fire coaches who happen to have losing records. Everyone is fooled that changing coaches has a larger effect on team outcomes because of regression to the mean. A team that wins 4 games one season is bound to improve next season for all kinds of reasons not related to the coach. Firing and replacing a losing coach doesn't appear to have much effect beyond what regression tells us would happen anyway.
The second factor is the illusion of control, in which people overestimate their own ability to control outcomes in life, good and bad. We naturally project this overconfidence onto others, believing they have similar levels of control over events. But how much control did John Harbaugh have over Billy Cundiff's missed 32-yd FG in the 2011 season's AFC Championship Game? Or the now-legendary misplay by Denver safety Rahim Moore in the game-tying bomb in the playoffs last season?
Third, there is a philosophical and even ideological bias toward attributing outcomes to great leaders. The prevailing ideology in academia and the media sees history as a long line of great leaders whose personal vision and actions drive world events. Who is credited with freeing the slaves? Is it the millions of soldiers who fought for the Union in the Civil War? Is it the millions of people who participated in the emancipation movement? No, it's one guy--Abraham Lincoln. The other side of the philosophical debate sees events as caused more from the bottom up--societal trends drive history. And the other side of the debate sees events as more caused by the top down. And it's the top-down worldview that currently reigns.
This narrative is personified in Tony Dungy. Dungy is a beloved former coach and champion. I consider myself a fan of his. But remember the story when he took over the Colts and finally won a Super Bowl for Peyton Manning? The narrative was that the Colts had great players but couldn't get it done until they had a great leader like Dungy to put them over the top. But there was also the matter of Dungy's previous job in Tampa Bay. The Buccaneers won the Super Bowl the year after replacing Dungy. The narrative then was, 'Well, they did it with Dungy's players.' In one case, it's the players that makes the difference. In the other case, it's the coach that makes the difference. Both narratives are diametrically at odds, yet live on in the very same person.
Another factor is an illusion generated by network effects at the college level. Great high school players and junior college transfers understandably congregate toward winning college teams, perpetuating and enhancing the superiority of those programs. The coaches at the top schools naturally gain stature, which in turn further perpetuates their ability to recruit top talent. It's a self-reinforcing, rich-get-richer system. Once a coach has enough recognition, he becomes a franchise unto himself.
Consider Nick Saban, who is widely regarded as an unrivaled great football coach. His pro team was decidedly mediocre in Miami, and it was worse his second year than his first. One explanation for the difference is that he happens to have some sort of mysterious knack for college coaching but not for pro coaching. But no one can put a finger on what it is that is so qualitatively different about the function of a coach at either level. The alternative theory is that coaching doesn't have a large effect in either case, and that it's the players that matter. And Saban simply has a huge advantage in player talent at Alabama that he could never have in Miami.
Some claim there are large differences among coaches in the skill of player evaluation. How often are we reminded of how Bill Belichick drafted Tom Brady in the 6th round of the draft? But we rarely hear about how he's not able to draft defensive backs. We hear the name Jeff Marriott even less often. Who's Jeff Marriott you ask? He's the defensive tackle New England chose with its 5th round pick the same year they took Brady. Marriott never played a snap in the NFL. If Belichick really had any inkling at all that Brady would be a solid QB, much less a Hall of Fame bound superstar, don't you think he might have used a measly 5th round pick to grab him, just to be safe? The truth is he thought Jeff Marriott wasn't worth risking leaving on the board until the 6th round, but Tom Brady was.
Don't get me wrong. I'm not saying Belichick is a poor talent evaluator. I'm claiming he's indistinguishable from most other coaches and GMs in that department. He has hits and misses just like everyone else. But because of his team's success, he has been bestowed an aura of greatness.
The importance of the coach is overrated in the NFL for a number of reasons--mathematical, statistical, psychological, and philosophical.