A Better QB Passer Rating

The QB passer rating system is roundly criticized for various reasons, yet it continues to be used in the media and by analysts around the NFL. The formula used to create the passer rating is a bizarre, complicated, redundant, incomplete, and arbitrary equation. And it should be replaced.

CURRENT NFL PASSER RATING CRITICISMS

It is bizarre because the result, the actual rating, signifies nothing. The units are in terms of...well nothing--not yards, or points, or completions, or their equivalents.

It is complicated because it combines four separate components that each practically require a slide ruler to compute. For example, the interception component is 2.375-(Int/Att x 25).

It is incomplete because it does not consider sacks.

It is redundant because it includes both completion percentage and yards per attempt. The yards per attempt stat is highly dependent on on completion percentage. Yards per attempt is actually just (yards per catch) x (completion %). Therefore, completion percentage is double counted in the formula.

It is arbitrary because each of the four components are not weighted in any meaningful way. The components of the formula are based on multipliers and constants selected to give the rating a nice scale rather than based on their importance to scoring or winning. They also use arbitrary maximums--each component is capped at 2.375 for an unknown reason. Look at the interception component formula above. Why is it multiplied by 25? Why not 30 or 20?

My personal criticism of the passer rating is that it includes touchdown passes. They should not be included in a passer rating because they are the result of many other factors beyond QB passing proficiency. For example, a QB could be on a team with a terrific defense that frequently produces turnovers deep in an opponent's territory. Or he could benefit from a great running offense that sustains drives. Further, TD passes are the culmination of all the other attributes of the passer. Accuracy, avoiding interceptions, and avoiding sacks all lead to TD passes.

I contend that passer attributes that enable the TD passes are far truer measures of passing performance than whether a WR broke a 7 yard slant pass for 50 yard TD run. Why should the QB get credit for that? What about a perfectly thrown 40 yard pass to a WR knocked out of bounds at the 1 yard line, followed by a 1-yd TD run? We don't need a fantasy football scoring formula, we need a reliable passer rating system that tells us how much a quarterback contributes to his team's success.

BUILDING A BETTER RATING SYSTEM

My goal is to create a meaningful passer rating based on factors controlled primarily by the quarterback alone. The weights of each component are based on their statistically predictive power of team wins. The units of the rating are in "wins added." In other words, the result of the new rating reveals, holding all other team performance measures equal, how many wins above (or below) average, a QB's passing performance would add to his team over the course of 16 games.

The formula is based on a multivariate regression model of team wins. Using data from the past five NFL regular seasons, the regression model estimates team wins based on the efficiency stats of each team including passing, running, turnovers, and penalties. Regression models can hold all other factors equal. By only adjusting the factors of interest (passing performance) we can see the effect on the estimate of season wins. The regression model automatically weights each factor corresponding to its actual importance in terms of team wins. Arbitrary weighting is not necessary.

The "wins added" model of rating athletic performance is based on a rich foundation of previous work in other sports. In baseball, Bill James's runs created statistic weights its various components in a way that approximates a batter's share of runs scored by his team. The recent book Wages of Wins proposed a similar stat for NBA basketball called Wins per 48 Minutes (WP48). The author, David Berri, weighted stats such as points scored, steals, turnovers, and assists based on a regression model of team wins. WP48 estimates each player's share of his team's wins. [I should also note Berri also attempted something similar with QB ratings, called QB Score, with far less success. It weights the components based on a regression and includes sack data. But it contains the same comlexities, redundancies, and other problems with traditional passer rating. In fact, it produces nearly identical results with the traditional rating, correlating at 0.95.]

The new rating system is also a derivative of a baseball stat called the Defense Independent Pitching Statistic (DIPS) developed by amateur sabermetrician Voros McCracken, now employed by the Red Sox. DIPS is a revolutionary concept because it intentionally ignores a large part of conventional pitching statistics. It is based on the idea that once a batter puts a ball into play, the pitcher has no control over whether it is a hit or an out. It is mostly due to luck, defensive positioning, and fielding skills. DIPS considers only those stats that a pitcher has total control over such as strike outs, walks, and home runs. Although DIPS only includes a subset of pitching performance, it is actually more predictive of a pitcher's year-to-year ERA than his actual ERA. It's better because it's a truer measure of pitching ability that ignores the noise of luck and other factors beyond the pitcher's control. The stats that remain are good proxy variables for the ones not included.

This new passer rating system is based on the same concepts as runs created, WP48, and DIPS. Like runs created and WP48, the new passer rating system is computed in terms of a concrete outcome. In this case, it is in terms of team wins instead of team scoring. And similar to DIPS, the new system includes performance measures primarily controlled by the quarterback. Although it is much harder in football than baseball to extricate one player's contribution from the rest of his team, the new system ignores spurious stats such as TD passes.

In summary, the new passer rating:

1. Is not arbitrary. Each component is weighted exactly as much as their relative importance to winning games. These weights are derived from a regression model using data from all teams since the 2002 expansion.

2. Is computed in terms of team wins. The regression model allows the passer rating model's component weights to translate directly into how many additional wins a QB's performance would yield, on average, over 16 full games.

3. Is not redundant. The components do not double count passing stats.

4. Includes only the passing stats primarily controlled by the QB. Factors such as passing yards after catch are not included. Admittedly, this is an imperfect feature of this rating system, but that is due to the team nature of the sport of football. No method could perfectly separate a single player's contribution from his team's.

The three components of the new system are completion percentage, interception rate, and sack yardage rate. In three prior posts, I discuss why I included each one, how they are calculated, and list how the QBs of 2006 rank for each component. The resulting formula of the new passer rating is:

QB Wins Added = (Comp% * 0.18) - (Int/Att * 50.5) - (Sack Yds/Att * 1.57) - 8

Why subtract 8? Because that is the average number of wins achieved by the average QB, and we're interested in 'wins added' above the average performance.

2006 PASSER RATINGS

Now let's look at how the QBs of 2006 fared in the new passer rating system. The total 'wins added' for each quarterback is listed. The 'wins added' for each component of the rating are also listed, along with how each QB ranks among his peers. The traditional NFL passer rating (QBR) is also listed for comparison.


RankPlayer+Cmp% WinsCmp% Rank+Int WinsInt Rank+Sk WinsSk Rank+Total WinsQBR
1Manning1.2530.9550.5012.69101.0
2Brees1.1250.7790.4542.3496.2
3Garcia0.65151.2220.4152.2895.8
4Carr1.8410.4215-0.05292.2182.1
5Huard0.47201.5510.09222.1198.0
6Brunell0.76120.9940.20171.9586.5
7McNair0.8980.49140.4631.8482.5
8Bulger0.8791.083-0.17371.7892.9
9Rivers0.65160.7970.27121.7192.0
10Rattay0.42210.7780.4621.6688.2
11Pennington1.1640.14230.21151.5182.6
12Brady0.67140.61110.23131.5087.9
13Romo1.302-0.12320.19191.3795.1
14Palmer0.76130.52120.08251.3693.9
15Warner1.1260.3020-0.16351.2589.3
16Delhomme0.53190.50130.16201.1882.6
17Leftwich0.17260.41160.3470.9279.0
18Johnson0.62170.09250.06270.7672.0
19McNabb-0.19320.8160.08240.7095.5
20Losman0.80100.1622-0.36410.5984.9
21Frye1.127-0.3438-0.21400.5772.2
22Kitna0.7811-0.0430-0.19390.5479.9
23Favre-0.37360.31190.4060.3472.7
24Garrard0.3823-0.06310.02280.3480.5
25Culpepper0.42220.6510-0.78450.2877.0
26Manning-0.07290.07260.20180.2077.0
27Leinart-0.23330.20210.11210.0874.0
28Smith0.0028-0.01270.07260.0674.8
29Harrington-0.1030-0.12330.28100.0568.2
30Cutler0.1825-0.0229-0.16340.0088.5
31Green0.5418-0.4439-0.1636-0.0674.1
32Campbell-0.90430.33180.338-0.2376.5
33Gradkowski-0.73400.41170.0823-0.2465.9
34Roethlisberger0.2924-0.6142-0.1232-0.4375.4
35Plummer-0.5237-0.23350.2114-0.5468.8
36Grossman-0.6339-0.27370.299-0.6073.9
37Hasselbeck-0.2734-0.2134-0.1533-0.6376.0
38Wallace0.0227-0.6343-0.1131-0.7276.2
39Brooks-0.1431-0.2736-0.5244-0.9261.7
40Young-1.1845-0.02280.2016-1.0066.7
41Vick-0.99440.1224-0.3642-1.2375.7
42Bledsoe-0.8641-0.5341-0.1738-1.5669.2
43Simms-0.6138-1.37440.2811-1.7146.3
44Anderson-0.3035-1.4845-0.0930-1.8763.1
45Walter-0.8642-0.5240-0.5143-1.8955.8

We see that Manning, Brees, and Garcia top the list, but some surprises follow. QBs such as David Carr, Mark Brunell, and Tim Rattay actually fare pretty well. Each of those three QBs place in the top 10 for different reasons, as you can see by how they rank in each component. Those QBs performed fairly well despite being surrounded by below-average teams.

McNair ranks high at number 7. Although he wasn't starting on anyone's fantasy team last year (16 TDs all year), he led Baltimore to 13 wins by avoiding sacks and keeping the ball out of the hands of the opponent's defense.

One final note: I think this formula is incomplete. You might notice that QBs who tend to throw short passes tend to rank highly. These QBs tend to have good completion % and interception stats. The formula should include this consideration. I think the best way would be to use yards per completion minus yards after catch. The stat would be called "air yards" or something catchy. Unfortunately, YAC totals for quarterbacks are not publicly available--I've searched everywhere. I know it is kept by Stats, Inc., but I can't find it for any year after 2002. If anyone knows where to find QB YAC stats, please leave a comment.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

15 Responses to “A Better QB Passer Rating”

  1. Derek says:

    Very interesting use of the regression model. Perhaps you could use yards per completion in addition to completion percentage so that QBs like David Carr get penalized for throwing a lot of underneath passes. At least you wouldn't have as much of an overlap in the 2 variables that way.

    I find it interesting how Culpepper played OK in terms of interception rate and completion % (small sample size probably plays a factor), but his immobility was such a huge hindrance.

    You could also adjust completion percentage by drops, balls thrown away, spikes, etc. These numbers seem more reliant on the receiver than the QB. Although underthrows/overthrows depend on WRs being able to run their routes properly, the QB's accuracy is just as important.

    Still, I think it's nearly impossible to separate the performance of the QB from the rest of the offense. The only way to say with some certainty how many wins a QB actually contributes is to see how backups do in meaningful situations by comparison. But then you run into sample size issues, etc.

    I like the +/- statistic for basketball myself because that can be normalized for teammates playing at the same time as the player being rated.

  2. Brian Burke says:

    Culpepper may have tried to scramble the same way he did prior to his injury. He still has all his passing skills, so he could still play effectively if he can adapt to being a pure pocket passer, using the RB outlet pass instead of scrambling to beat the rush.

  3. coldbikemessenger says:

    thanks, now it looks much better.
    Have you considered adding some sort of tards per att component?

  4. Dan says:

    I'm just wonder how well a linear (as opposed to non-linear) model fits the data. Did you check the residuals to see how they look?

  5. Brian Burke says:

    Yes, the residuals looked pretty lopsided when I used a straight OLS model, but I ended up using a heteroskedasticity-corrected weighted model and the residuals were even and random-looking. The win projections also became more accurate.

  6. MRH says:

    Re sack rate = sacks/pass attempts.

    I think sack rate should = sacks/(pass attempts + sacks),
    that is the rate of sacks should be calculated by the number of times dropping back to pass. I realize that scrambles where a QB drops back to pass and then runs for a gain are excluded in this system.

  7. Brian Burke says:

    Regarding attempts: I do count sacks as "pass attempts" for determining sack rate, just as mrh suggests.

  8. Dan says:

    So what was causing the heteroskedasticity and what gets weighted to correct for it?

  9. Brian Burke says:

    Dan-I use Gretl stat software. It has a feature that automatically corrects for heteroskedasticity in OLS models. The variance of the dependent variable, is not constant across all 17 possible w-l records. It gets bigger towards more wins. This results in a bad pattern of residuals. To correct, I believe the software applies weights to the dependent variable across the range. There is a weighted OLS option as well, where you can manually input weights, but that's not what I used. I'm on vacation this week, without my software, so I can't give you a very good answer for the moment. Thanks for keeping me honest.

  10. Anonymous says:

    snort.....tards per attempt. that's great.

  11. Brian Burke says:

    Dan-I was mistaken about which OLS model I used. For this model (efficiency-win), I used a straight-up OLS and the residuls were nicely distributed. I had used the weighted model for another study I did comparing the continuity of wins from year to year for each team.

  12. Anonymous says:

    David:
    Do HS passing stats include yards after catch

  13. Don P. says:

    Please allow me to offer my congratulations to the author of the above rating system for offering a reasonable alternative to the NFL’s needlessly complicated rating system.

    Football fans need an efficiency measure similar to baseball’s batting average because fans of that game understand how that metric is calculated. If fans understand how a metric is calculated, they can easily grasp the more important question of why it is calculated a particular way. The batting average, for example, offers baseball fans a broadly meaningful way to comparatively evaluate player performance. That explains why baseball fans study and discuss batting averages so widely. Football has no equivalent because so few understand how the passer efficiency rating is calculated. Thankfully, the system described above solves that problem because of its sheer simplicity.

    Please allow me to offer an observation: Eddie Epstein, a contributor to ESPN.com, has conducted exhaustive correlation studies of a vast range of NFL metrics. His studies show that the metric that correlates most closely with winning is the difference between yards gained per pass attempt and yards allowed per pass attempt. It seems reasonable, therefore, that any passer efficiency measure should be based primarily on yards gained per pass attempt. In addition, I agree with the author presenting here that the TD ratio captures effects extraneous to passer efficiency per se, and therefore, should not be included in any passer efficiency rating system. Excluding it also has the advantage of simplifying the rating system. I did mention the whole simple-is-better thing, didn’t I? Yes, I did.

  14. Anonymous says:

    This model fails to take into account the changing nature of the game. Ten years from now, the rules and general direction of the game may cause certain stats to have greater weight toward winning.

    In order for the model to work, the weighting system of it must be changeable over a sufficient time interval. Perhaps changing it every season is too much, but it needs to be done. Because the game changes.

  15. Anonymous says:

    this is way old, but I would like to say that this part -- "The yards per attempt stat is highly dependent on on completion percentage. Yards per attempt is actually just (yards per catch) x (completion %). Therefore, completion percentage is double counted in the formula." -- is missing the point of including YPA. alright, in some ways it double counts completion percentage. however, the reason it's there is because there is a difference in completing 50% of your passes for an average of five yards each than 50% of your passes for an average of ten yards each. a QB that's 10/10 for 30 yards isn't any good. I'd say yards per catch needs to be incorporated into any system and because of that, yards per attempt is probably more important than completion percentage.

Leave a Reply