Signal vs. Noise in Football Stats

In 2007, the Detroit Lion defense began the first half of the season with 13 interceptions, the most in the NFL. The next best teams had 11. It's reasonable to expect that the Lions would tend to continue to generate high numbers of interceptions through the rest of the season, notwithstanding calamitous injuries.

I wouldn't expect them to necessarily continue to be #1 in the league, but I'd expect them to be near the top. And I'd be wrong. It turns out they only had 4 interceptions in their final 8 games, ranking dead last. So halfway through the season, if I were trying to estimate how good the Lions are in terms of how likely they are to win future games, I might be better off ignoring defensive interceptions.

Although turnovers are critical in explaining the outcomes of NFL games, defensive interceptions are nearly all noise and no signal. Over the past two years, defensive interceptions from the first half to the second half of a season correlate at only 0.08. In comparison, offensive interceptions correlate at 0.27. As important as interceptions are in winning, a prediction model should actually ignore a team's past record of defensive interceptions.

You might say that if defensive interception stats are adjusted for opponents' interceptions thrown, then the correlation would be slightly higher. I'd agree--but that's the point. Interceptions have everything to do with who is throwing, and almost nothing to do with the defense.

This may be important for a couple reasons. First, our estimations of how good a defense is should no longer rest on how many interceptions they generate. Second, interception stats are probably overvalued when rating pass defenders, both free-agents and draft prospects.

I've made this point about interceptions before when I looked at intra-season auto-correlations of various team stats. That's a fancy way of saying how consistent is a stat with itself during the course of a season. The more consistent a stat is, the more likely it is due to a repeatable skill or ability. The less consistent it is, the more likely the stat is due to unique circumstances or merely random luck.

The table below lists various team stats and their self-correlation, i.e. how well they correlate between the first half and second half of a season. The higher the correlation, the more consistent the stat and the more it is a repeatable skill useful for predicting future performance. The lower the correlation, the more it is due to randomness.
















VariableCorrelation
D Int Rate0.08
D Pass0.29
D Run0.44
D Sack Rate0.24
O 3D Rate0.43
O Fumble Rate0.48
O Int Rate0.27
O Pass0.58
O Run0.56
O Sack Rate0.26
Penalty Rate0.58

In a related post, I made the case that although 3rd down percentage tended to be consistent during a season (0.43 auto-correlation), other stats such as offensive pass efficiency and sack rate were even more predictive of 3rd down percentage. In other words, first-half-season pass efficiency predicted second-half-season 3rd down percentage better than first-half-season 3rd down percentage itself.

But what about other stats? Are there other examples where another stat is more predictive of of something than that something itself? Below is a table of various team stats from the second half of a season and how well they are predicted by other stats from the first half of a season.

For example, take offensive interception rates (O Int). Offensive sack rates (O Sack) from the first 8 games of a season actually predict offensive interception rates from the following 8 games slightly better than offensive interception rates (0.28 vs. 0.27).








































PredictingWithCorrelation
D FumD Fum0.33
D FumD Sack 0.15
D FumD Run0.12
D Int D Sack 0.08
D Int D Int 0.08
D Int D Pass0.01
D PassD Pass0.28
D PassD Sack 0.26
D RunD Run0.44
D Sack D Sack 0.24
D Sack D Pass-0.07
O 3D PctO Sack -0.53
O 3D PctO 3D Pct0.43
O 3D PctO Int -0.42
O 3D PctO Pass0.42
O 3D PctO Run0.08
O FumO Fum0.48
O FumO Sack 0.24
O Int O Sack 0.28
O Int O Int 0.27
O Int O Run0.06
O Int O Pass-0.37
O PassO Pass0.49
O PassO Sack -0.33
O PassO Run-0.10
O RunO Run0.56
O RunO Pass0.00
O Sack O Pass-0.40
O Sack O Sack 0.26
O Sack O Run0.03
PenPen0.58
PenD Pass-0.23
PenO Sack -0.08


There are a thousand observations from this table. I still see new and interesting implications whenever I look it over.
  • Having a potent running game does not prevent sacks.
  • The pass rush predicts defensive pass efficiency as well as defensive pass efficiency itself.
  • Running does not "set up" the pass, and passing does not "set up" the run. They are likely independent abilities.
  • Offensive sack rates are much better predicted by offensive passing ability than previous sack rates.
  • Defensive sack rate predicts defensive passing efficiency, but defensive passing efficiency does not predict sack rate.
We see that many stats, such as passing and running efficiency predict themselves fairly well. But even those stats might be better predicted by using a combination of themselves and related stats. For example, in my previous post I noted how accurately offensive 3rd down percentage could be predicted using passing efficiency, sack rate, and interception rate.

The implications of these auto-correlations are numerous. Team "power" rankings and game predictions (both straight-up and against the spread) rely on a very simple premise--past performance predicts future performance. We now know that's not necessarily true for some aspects of football.

Lions head coach Rod Marinelli might be banging his head against the wall trying to understand how his defense was able to grab 13 interceptions through game 8, but only 4 more for the rest of the season. He's wasting his time. The answer is that in the first half of the season, the Lions played against QBs Josh McCown (2 Ints), Tavaris Jackson (4 Ints), and Brian Griese twice (4, 3 Ints).

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

9 Responses to “Signal vs. Noise in Football Stats”

  1. dfobare says:

    Over how many seasons were these correlations run?

    re:Lions ints

    The drop in the 2nd half of 2007 was likely caused by the collapse of Big Baby, Shaun Rogers.

  2. Brian Burke says:

    2006 and 2007 (n=64).

  3. Mr.Ceraldi says:

    Hi Brian...this is a great post (even when looking at it for a second time)..
    I have a stat calculation question?
    how do you combine two stats together?
    that seem to have different units and keep the same ratios? when I try it
    it messes up..

    example "offensive 3rd down percentage could be predicted using passing efficiency, sack rate, and interception rate"
    would you mind walking through one example for
    a novice statitiscian..thanks dan

  4. Brian Burke says:

    Dan-I don't have the exact method handy, but what I did was do a linear regression with 3rd down % from the 2nd half of the season as the dependent variable, and pass efficiency, sack rate, and interception rate from the 1st half of the season as independent variables. This regression actually predicts 2nd half-season 3rd down % better than 1st half season 3rd down %.

    The units aren't important. In a linear regression the generated coefficients are always in "units of dep variable per unit of indep variable." So the ultimate prediction from the model will always be simply in terms of the dependent variable.

  5. Mr.Ceraldi says:

    Brian one further question
    your model is based on these rates
    Offensive pass efficiency, including sack yardage
    Defensive pass efficency, including sack yardage
    "Offensive run efficiency
    Defensive run efficiency
    Offensive interception rate
    Defensive interception rate
    Offensive fumble rate
    Penalty rate (penalty yards per play)"

    how did you select these? did you base it on
    correlations of each stat to team wins ? Or are these the stats with the higest % from the non-linear
    logit regression? just curiuos def int rate
    was left out due to its randomness what did the logit regression show for it?
    thx again dan

  6. Brian Burke says:

    ,I picked them based on 3 considerations--

    1. Each variable had to be independent from the others (or "orthogonal" as statisticians say). That is, one variable couldn't be correlated with the others.
    2. Each variable had to be predictive of wins. That means it has to (a) correlate well with winning.
    3. And (b) the direction of causation had to be clear. Total rushing yards, for example, correlate highly with winning, but it's teams that are already ahead that rack up extra rushing yards in the 4th qtr.

    Special teams stats were not significant.

    This post shows the logit coefficients with def ints left in.

  7. Mr.Ceraldi says:

    brian thanks for getting back so quick!..this helps a couple follow-ups ...

    "The units aren't important. In a linear regression the generated coefficients are always in "units of dep variable per unit of indep variable." So the ultimate prediction from the model will always be simply in terms of the dependent variable."

    is the same true for logit regression?
    does the specific rate you used for each of
    your stats matter i.e opass is yards per play penalty rate I'm assuming is penalty yards? per game

    A theory question...if we were to find another stat that fit your criteria above (independent, correlated to winning etc) at say .25 or .35 would it neccessarily improve your model?? are correlations cumulative in this way..in other words if we hypothetically found another magical stat that fit your criteria above
    ( was independent etc etc) would it neccessarily improve your predictions?
    One last thing..
    I am building a model for another sport ( ice hockey) and wanted to know your suggestion for cutoff values for inclusion in a model..what would you consider a strong enough correlation value for a stat to be considered skill vs random luck (I know .08 is random) but is (.25)strong enough to be consideredrepeatable and skill? as you know ice hockey has much more randomness in it which i'm struggling to work through)
    Dan

  8. Brian Burke says:

    In logit regression, the units are weird. It's "change in the logarithm of the odds ratio per unit of independent variable." But effectively, it's the same. The regression is measuring normalized variance, so units are transparent.

    No, there is no official cut-off for correlation. I'd use it as a guide, though. I'd start with everything that you believe has a logical cause-effect relationship with your dependent variable, as long as everything is reasonable independent. For example, don't use penalty minutes and penalties in your model, because they're both measuring the same thing.

    Run the regression and note how well it predicts results. You can then remove or add variables and see how well it improves or hurts the model.

  9. Roger says:

    Do you think correlating first 8 games with last 8 games is the best way? Changes in weather and playoff considerations (among other things) could affect the various efficiency. To truly test the correlation of the various variables with each other you should be randomizing your two sets by choosing 8 random games for the first variable and the remaining 8 games for the second variable. And ideally do that over the 16 choose 8 possible groupings (or a statistically large enough sample of those 12870 combinations.)

Leave a Reply