Fighter Pilots and Firing Coaches

At its heart, football strategy is game theory. But at the heart of game theory is utility theory--the science of defining and measuring what is useful. Utility in football is tricky. A 4 yard gain on 3rd and 5 isn't very helpful, but the same gain on 2nd and 4 is.

Utility theory is also at the heart of economics, which may be why we see some coaches with economics backgrounds do well. One of the cornerstones of utility is Prospect theory, a concept developed by researchers Daniel Kahneman and Amos Tversky. Simply stated, Prospect theory says that people are averse to losses more than they value equivalent gains. It was a major breakthrough, and it won the 2002 Nobel prize for economics.

But this article isn't about Prospect theory as much as it is about some of Kahneman's earliest work. While Tversky was always an economist, Kahneman's training was as a psychologist.

In the late 60s, Kahneman was a consultant for the Israeli Air Force. He lectured instructor pilots on the latest research that showed that reward was far more effective than punishment at improving performance. Instructor pilots were not buying any of it.

They told Kahneman, when student pilots have a bad flight we yell and scream at them, and the next day they tend to do better. But when they have a good flight we'll praise them like you suggest, and they tend to do worse.

It was then that Kahneman realized how natural variation in performance, and its natural regression to the mean, were fooling the flight instructors into believing it was their yelling and screaming that improved the student pilots' performance. Flight instructors saw this:

student has poor flight --> yelling and screaming --> improvement on subsequent flight

When in reality, they were just as likely to see this:

student has poor flight --> improvement on subsequent flight

What does this have to do with firing coaches? Owners rarely fire coaches who just finished a successful season. A new coach comes in and there is usually an improvement, but that improvement would tend to happen anyway, with or without a new coach, simply due to regression to the mean.

For those who aren't familiar, regression to the mean is not metaphysical mumbo-jumbo. It just means that the change in performance from one day to the next is most often in the direction of the overall average performance (and not directly to the average itself, or to the average and no further). It occurs because when things go really bad it's usually because several bad things all happened at once to spoil the outcome. And when things go really well it's usually because several good factors conspired to win the day. It's not likely that all those factors will repeat in the same way.

Regression to the mean is not a rule, but it is a law. In other words, in some cases good outcomes get even better the next time, or sometimes bad outcomes get even worse. But the fact that most of the time good outcomes won't be as good and that bad outcomes won't stay as bad is an unalterable fact of life.

For example, when the 2008 Detroit Lions went 0-16, it was because they had injuries at critical positions, little talent on defense, a tough schedule, plus the ball never really bounced their way. Next year, it's unlikely that all those things will occur again, and they'll likely win a handful of games. And when the 2007 Patriots went 16-0, several factors all went their way. They had acquired lots of talent on both sides of the ball, had a relatively easy schedule, and stayed healthy all year. In 2008, they didn't stay healthy and mustered 5 fewer wins.

NFL owners and team executives, repeatedly witness this cycle:

team does poorly --> fires coach --> team does better

And they are tricked into attributing the team's improvement to the firing of the coach. The result is a belief that replacing the coach is more helpful than it truly is. In reality, this is almost as likely to happen:

team does poorly --> team does better

In fact, how do we know that starting over with a new coach doesn't hurt more often than it helps? Perhaps teams with new coaches improve slightly less than similar teams who keep their coaches. I'm not claiming that all coaches should keep their jobs no matter how poorly their team does. My point is that replacing the head coach is probably not nearly as effective a move as it appears.

The recent parity of team talent levels in the NFL has made the regression effect more prominent. Because teams are generally closer to each other in talent, it is much easier to go from few wins one year to many more the next (and vice versa). Only a slight improvement or decline is required to leverage a large change in a team's fortunes. This may be why the average coaching tenure is getting shorter.

This lesson should extend into all sports, and even day-to-day life. Beware of stories like these: "I had several bad colds last year, then I started taking [herbal product X], and now I don't get sick as often." Or, "My portfolio took a beating last quarter, so I changed my fund manager, and now my investments are making gains again." How about, "Our economy was in a slump, so we borrowed and spent $900 billion to cause it to recover." Or maybe, "My NFL franchise lost too many games last year, so I brought in Bill Parcells, and now it's doing much better."

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

21 Responses to “Fighter Pilots and Firing Coaches”

  1. Steve H says:

    Not only do people ignore the regression to the mean (signal+noise is regarded as all signal), but there's also the action bias. It basically says that doing something is better than doing nothing, even if the expected result is worse. There's a paper on it applied to soccer goalies here: http://mpra.ub.uni-muenchen.de/4477/

    Then again, maybe GMs know all this, and there's a principal-agent issue and the GMs want to look smart and fire coaches based on a high negative deviation. Then the firing of the old coach, and the hiring of the new coach, will look like the right call and the GM builds job security.

  2. Ian says:

    Ahh, speed camera site logic. We had a crash here -> we put a camera there -> we didn't have another crash there. Therefore the camera must be working.

    Or if you're a Simpsons fan, there was an episode where Lisa sold Homer a rock, on the grounds that it keeps tigers away.

  3. Brian Burke says:

    Or in DC, it's more like:

    crash here-->put camera here-->revenues go up

  4. Ian says:

    Steve H -

    A very interesting paper, thanks for the link. Action bias certainly seems an interesting theory to look into given the current economic climate. Politicians are choosing action over inaction without being certain over whether it will help or hinder, basically because it looks better.

    Rules of the politics game are
    Action + success = successful action
    Action + no success = well, at least we tried
    Inaction + no success = why aren't you trying to help
    Inaction + success = Lucky

    In terms of firing coaches, the same rules apply. Clearly, the fallacy is that inaction cannot lead to success.

  5. Steve H says:

    Ian - Look at Nixon and inflation. All his economic advisers tell him price controls are a horrible idea, Nixon announces his new price controls, and he gets reelected because he was "doing something." I see the same thing now with the housing market.

  6. Doug says:

    [Sorry if this was posted twice (or thrice)]

    Ancient but relevant post from pfr blog:

    http://www.pro-football-reference.com/blog/?p=62

  7. Brian Burke says:

    Thanks, Doug. Added the link above.

  8. Phil says:

    Was about to say that should really be an easy thing to measure

    the link given above I guess does so, seems to suggest that coaches aren't being fired too quickly as teams changing coaches improvwe more frequently at 4 of the 5 win levels

    I think you're post make total sense in theory, I would have guessed that the evidence in the article would support your theory, that coaches staying on would have better records than new coaches , players would be more used to the system, coaches would have better sense of how to use players, etc.

    doesn't seem like the evidence support that though

  9. Phil says:

    another thought, maybe the article just goes to show how generic and similar most NFL coaches actually are, that they can be changed and players can pick up the new systems as quickly as they do

  10. Brian Burke says:

    Keep in mind that my point was changing coaches appears much more helpful than it really is, not that it's not helpful at all.

    Also, the table in Doug's post shows only 2 of 4 categories show any advantage in replacing the coach. (The 5th category does not have enough cases to make any conclusions.) There are only 16 cases of teams that changed coaches in each of those two categories, so any statistical inference is going to be on shaky ground.

    My hunch is that replacing coaches can be helpful, and Doug's data points in that direction. But unless people are aware of the regression to the mean phenomenon, they'll think replacing the coach is far more effective than it really is.

  11. Phil says:

    don't disagree with any of that

    after I read the post and thought about it, especially living in Pittsburgh and seeing all the good that have come from Steelers continuity, I hypothosized that teams keeping the same coach might improve at a better rate than teams changing coaches and was kind of mildly suprised by the article linked above

  12. Brian Burke says:

    Yeah, I was kind of hoping to see that continuity is actually a benefit--that switching coaches would usually be a slight setback, accounting for the regression effect.

    However, I think I've found a problem in Doug's analysis. He grouped teams by season wins, 0-3 wins, 4-5 wins, 6-7 wins, etc. The problem is that there is going to be a bias toward the lower end of each group.

    For example, take the 4-5 win category. We'd expect that teams with 4 wins are more likely to replace coaches than teams with 5 wins. And it's easier for a 4-win team to improve than a 5-win team, because of the regression effect discussed in the article. So within the 4-5 win group, we should expect to see teams that replace coaches improve more than those that don't, simply due to the bias.

    The same goes for the other categories. Zero, 1, or 2-win teams are more likely to replace coaches than 3-win teams. The lower your win total, the easier it is to improve.

    What I'd suggest is to abandon the category groupings. Make a scatterplot of team's win totals vs. next year's improvement. Use a linear-best fit to measure the strength of the regression for both types of teams-those that fire the coach and those that keep him. This will eliminate the grouping bias and account for the regression effect. Then just compare the amount of improvement between the two groups.

  13. Doug says:

    I don't disagree with what you've said, Brian. My post was just meant to be a real quick look. If it came off (either there or here) as anything more than "here's something to think about," then that's not what I intended.

  14. Brian Burke says:

    Doug-I see it the same way. Was just trying to respond to Phil's points above and trying to be constructive.

  15. Odysseus says:

    A major factor with some new coaches is simply the level of effort they demand and expect from their players. Look at Mike Singletary this year with the 49ers: he demanded effort and hustle, and effort is still the most important regular season factor. The talent is there on every team to compete with every other team; teams struggle because their system is bad, their talent is poor, or their effort is poor.

    Look at Randy Moss to see how important effort is. It doesn't matter the players you have if they give crap effort.

  16. JDTapp says:

    In Dean Oliver's Basketball on Paper he briefly looks at a few teams and what their natural regression to the mean would be for the following season and makes a prediction (easier to do with an 82 game season). He then offers that perhaps we can judge a coach based on how many more/fewer games they won than that benchmark, taking into account any major offseason changes/injuries/etc.

  17. jgordon1 says:

    The same thing can happen with a head coach and personell. Player A doing poor.. bring in player B. A head Coach in the NFL makes a mistake.. he gets fired and keeps millions... Wing Commander makes a mistake... Pilots die.. I would say that is a good incentive to learn

  18. Anonymous says:

    Leonard Mlodokov recently published The Drunkard's Walk. Although the book deals with the historical development of statistics he does deal with this phenomena. He writes of the perception that coaches who yell at players engender improvement in performance, that fund managers and CEOs get credit in many situations where their actions/decisions were not responsible for the results with which they are credited. I offer it here in support of this argument and as a great read.


    Ed

  19. Brian Burke says:

    That's where I got the idea for the post. The book has a section on the Kahneman story about the IAF.

  20. bytebodger says:

    I agree entirely with your analysis, but too often the democratic effects of capitalism provide a perverting influence. NFL owners (like fund managers and politicians) can't simply be content to take the most statistically-valid course of action. If they make the statistically "correct" choice (e.g., they keep the coach) and the results are poor (the team has another bad year), they face the very real fear that ticket sales fall.

    In the stock market, a company that misses its estimates (but misses them by LESS than was expected) sees its stock rise. Whereas a company that beats its estimates (but beats them by LESS than was expected) sees its stock fall. A similar scenario plays out in sports.

    I live in Jacksonville and if we had a rookie head coach the fan base would probably be OK with a 6-8 win season. Now that Del Rio's team has badly missed expectations this year, I doubt that he will be afforded the same luxury if the team chalks up only 6-8 wins (or less).

  21. Anonymous says:

    Nice article. I also think that while there is likely an advantage to continuity, there are three issues that would have to be addressed in advancing the discussion:

    1. is there a means of selecting the RIGHT coach to a) deserve continuity, and b) improve the team more than the regression to the mean alone would have?

    2. staff composition can be as important as the name at the top; take away many of the assistants on the staffs of successful "new" coaches (for example, Mularkey in ATL or Ryan in BAL in 2008) and that might negate the positive effect beyond regression.

    3. draft picks. teams that do poorly also tend to have higher draft picks. And while one can point to teams like the Steelers, Patriots, and Colts that have flourished in recent years without picking higher than #20 regularly, the influx of talent off a "bad" season may be greater for the incoming coach (which, in addition to the "genius" of Jim Scwartz per Belichik and natural regression, may account for Detroit's improvement in 2009).

    Just a few thoughts. Thanks for the post.
    Greg

Leave a Reply