MVP Candidates Through the Lens of WPA

One of the cool things to do with a Win Probability model is to look at the total Win Probability Added (WPA) of a player. Each play in a game changes a team's chances of winning either positively or negatively, and the WPA for the plays in which a player takes part can be tallied. WPA discounts 'trash-time' performance, and emphasizes 'clutch' performance. The result roughly measures the past value of a player in terms of what matters most--winning games.

WPA is a narrative statistic. It tells the story of how a player performed, but it does not predict how he'll likely play in the future. It includes all the flukes, miracles, and everything else that makes football unpredictable and fun to watch. In essence, WPA is what the MVP award is all about.

I've tallied the WPA for the notable MVP candidates through Week 12 of the 2009 season. The table below lists various players and their WPA expressed a few different ways. There is total WPA, which is simply a raw sum. There is also WPA per game and WPA per play, which tell a slightly different story.


Player
Plays
Total WPA
WPA/G
WPA/Play
B.Favre
416
2.4
0.20
0.0058
D.Brees
392
2.3
0.19
0.0059
C.Johnson
276
1.2
0.10
0.0044
T.Brady
497
1.1
0.09
0.0022
R.Wayne
123
0.9
0.08
0.0072
P.Manning
471
0.8
0.07
0.0017
C.Benson
194
0.2
0.02
0.0009
A.Johnson
118
-0.2
-0.01
-0.0013
J.Allen
54
-0.3
-0.02
-0.0050
A.Rodgers
495
-0.3
-0.03
-0.0006
M.Jones-Drew
285
-0.6
-0.05
-0.0022
M.Schaub
477
-1.4
-0.12
-0.0029
A.Peterson
272
-2.1
-0.17
-0.0077

Right now, according to total WPA Brett Favre is the league MVP, but Drew Brees is in a close second place. Frankly, I'm very surprised. I thought it was the Vikings' defense and Adrian Peterson who were carrying the load for Minnesota, but it's really Favre who's mattered most. A healthy chunk of Favre's 2.4 is from his last second heroic touchdown throw against the 49ers to win the game.

Peterson's negative value is shocking. I double checked it to be sure, and also looked up a few other RBs for comparison. For example, Ray Rice in Baltimore had a total WPA of -1.7, Marion Barber had a WPA of 0.2, and Joseph Addai had a WPA of -0.3. Peterson's fumbles are certainly hurting him, but it's running in general that appears to be the problem. Most runs in the NFL may actually be setbacks, and except for their handful of breakaway runs almost all NFL running backs are indistinguishable--Peterson included. The exception this year is the Titans' Chris Johnson, who is the very definition of breakaway in 2009. He is third on my list with +1.2 WPA.

I'm also surprised by the bad numbers for the Texans' Matt Schaub and Andre Johnson. I guess they've been too inconsistent, particularly in high-leverage situations. I would guess much of their more traditional positive stats have come after a game has effectively been decided.

In terms of WPA per play (or more precisely "per mention"), it's Reggie Wayne who tops the list.  Receivers and other positions besides RB and QB don't have their number called as frequently, so it's very hard for them to accumulate a high total WPA. I should note that anytime the intended receiver was apparent on an incomplete pass, his name is mentioned in the play description, and the loss of WPA is charged to both the passer and intended receiver regardless of whose fault it was. But WPA is a narrative metric, so fair or unfair, this is how players are seen and valued.

The WPA system is most unfair to defenders. For example, if the other 10 Steeler defenders miss a tackle and Troy Polamalu is the last guy able to run down a ball carrier for a gain of 20 yards, that's a great play on Polamalu's part. But WPA only see's Polamalu's name mentioned in conjunction with giving up a 20-yard play. If Polamalu missed the tackle and gave up an 80-yard touchdown, his name doesn't even get mentioned, and he's not charged with the loss of WPA. So when valuing defenders, it is probably best to compare them within their position if at all. That is, compare Polamalu to other safeties or Jared Allen to other defensive ends.

Then again, maybe the WPA is most unfair to blockers. You never read a play-by-play description say "10-yard gain by 28-C.Johnson. Great block by 68-K.Mawae."

Unfortunately, at this point I can't just click the mouse to generate a list of the top players. I need to do a search for each specific player, then tally his WPA. So if there are any other players of note that you'd like me to look up, please mention it in the comments.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

43 Responses to “MVP Candidates Through the Lens of WPA”

  1. coldbikemessenger says:

    I would like to see
    Romo
    Miles Austin
    Witten
    Roy Williams
    Just for comparison's sake.

  2. coldbikemessenger says:

    I also think it would be really interesting to see a list of position groupings
    Like DL's vs DB's
    but i am guessing that would be insanely time consuming.
    I do think the list is very cool and am quite surprised by Peterson

  3. coldbikemessenger says:

    Sorry to keep posting by I think this is the coolest thing since the ipod.
    Another interesting thing would be to look at various team units in an upcoming game, like the playoffs.
    I think it would work really well with the effeciency charts you already do.
    I mean something like well this is the Saints defensive passing effeciency rating and they get most of their WPA's from the secondary while the Vikings get most of theirs from the DL.

  4. Michael says:

    As a Colts fan, I was initially surprised by Reggie Wayne's ranking, but in retrospect, it makes a lot of sense. He's been making big plays all year.

  5. Anonymous says:

    If possible, I'd really enjoy seeing the top-10 for every position in the league. Or even each team's top performer.

  6. Anonymous says:

    Can you do this for the whole decade? Count how many WPA Tom Brady, Brett Favre, and Peyton Manning added.

  7. Brian Burke says:

    Yes, I can do the whole decade. I'd like to make some improvements to the entire WP model before I really dig much deeper into player WPA.

    Right now, I don't have the process automated for all players. I have to do a search for a certain player, then sum all the WPA for his plays. Maybe it's something I can automate in the off-season.

    For the Cowboys, here are WPA totals:
    Romo: +.02
    Collie: -.11
    Witten: -.10
    Williams: -.07

    Keep in mind that for receivers, they are penalized for incomplete passes intended for them whether it's their fault or not.

  8. Eiad says:

    I'd love to see Vince Young's WPA. Has he really been the difference in the win streak or has the team played better?

  9. Brian Burke says:

    Eiad-Great question. I should make a entire article out of that one, but I won't make you wait for the answer. Vince Young seems like the real deal in '09.

    V. Young: +0.86 WPA, 183 plays, .0047/play, 5 games started, 0.17/start

    K. Collins: +0.04 WPA, 222 plays, .0002/play, 6 games started, 0.04/start

  10. Brian Burke says:

    That game-winning drive VY had last week where he converted 3 4th downs probably accounts for most if not all of that 0.86. Take away that drive and he's a negative. But hey, that drive counts, right?

  11. Michael says:

    How is "trash time" measured?

  12. Michael says:

    Also, are any of those #s statistically significant?

  13. alex says:

    Colts should target Garcon less and Wayne more imo

  14. Jwsinclair says:

    How about Tony Gonzalez?

    As a Falcons fan, most of the highlights of this (slightly disappointing) season have been key 3rd down completions to Gonzalez, typically with at least two defenders already clinging to him.

  15. Zach says:

    I'm surprised that Peyton is so low given the Colts' comebacks and their close games. I'd like to see Phillip Rivers and Aaron Rodgers.

    Your play-by-play data is automatically taken from NFL.com gamebooks, right? Will you ever release the code you use to grab the PBP data?

  16. Pete McCabe says:

    I'll second Philip Rivers. For as much success as the Chargers' have had so far this season with a modestly good defense and no running game to speak of, Rivers seems a more deserving MVP candidate than half this list.

    How about kickers? I would think place kickers would score absurdly high on the per-play rankings since half their kicks are scoring chances.

  17. Brian Burke says:

    P.Rivers is -0.20 total WPA. Weird. Aaron Rodgers is in the original list.

    Kickers would have a chance to score high in WPA, but remember that WP already factors in the league average chance to make a kick. So if a kicker makes an 18-yd FG to win a game, he gets a +0.01 WPA. If he makes a 50-yd FG, he'll get about a +0.50 WPA.

    Trash time is accounted for because once a game has been decided, even the biggest play will not move a team's WP. Even a 99-yd TD play will result in a relatively small WPA. But when the score is close, the same play will drastically change WP.

  18. billsfan says:

    It would be interesting to see the numbers for some of the big-play receivers, like DeSean Jackson, Miles Austin, Sidney Rice, and almost every Saints wideout, who have scored multiple 50+ yard touchdowns. None have as many catches or TDs as Wayne, but Meachem, for example, has 7 TDs on 21 catches for 402 yards, and should be a good candidate for the lead in WPA/play.

    Peterson's WPA is intriguing. Does most of that come from the fumbles? I'd think he'd otherwise have enough long runs to offset the negative value of short runs.

  19. DSMok1 says:

    Burke:

    How are you splitting credit for plays?

    I have completed research based on the 2005 play-by-play database that treats each QB, receiver, and defense as a separate variable. I calculated the different weights for each of those 3 variables (on each play) to best predict the play's outcome.

    Here were the weights I found:
    Receiver 44.5%
    QB 33.5%
    Defense 22%

    Now I will add that the OL is combined with the Receiver and QB portion of the weighting; I have not figured out a good way to split them out yet.

  20. Miles Libbey says:

    Hey Brian-- Might want to check out
    http://www.ninersnation.com/2009/12/4/1185753/singletarys-formula-for-success
    -- the niner's nation guys have a new Ball security metric that might be useful for your wpa model.

  21. Shaun says:

    as an interesting project i wouldnt mind seeing is if previous MVPs have deserved thier title according to this, for example i felt that last year Warner was more deserving than Manning.

  22. Joe G says:

    I'd like to see a "possession" receiver like Wes Welker.

    Also, what about FG kickers?

  23. Joe G says:

    For Farve, are those total percentage points, so his total is 240% ?

    So you could say so far he has been worth 2.4 wins?

  24. Joe G says:

    Is the interpretation that Farve has been worth an extra 2.4 wins over an average QB?

  25. Brian Burke says:

    The WPA is not split between players. A TD pass from Favre to Rice might net +0.10 WPA for the thee play. Both players get credited with 0.10 WPA.

    Joe G-Yes. 2.4 means 240%. I'm not sure if that means 2.4 wins or 4.8 wins, to be honest. Each team starts with a 0.50 WP, so you only need to net 0.50 WPA to claim a win.

  26. Brian Burke says:

    Some more receivers, in plays, WPA/play, WPA total:

    W.Welker 149, .0031, +0.46
    R.Moss 120, .0050, +0.59
    M.Colston 80, .0097*, +0.78
    D.Henderson 59, .0024, +0.14
    J.Shockey 61, .0029, +0.18
    R.Meachem 35, .0130**, +0.45

    Colston and Meachem would top R.Wayne as the per play leaders. Meachem's numbers are insane.

    I've also been scrutinizing Adrian Peterson's numbers in detail. I think there is a bias in my system against short scoring plays, which would penalize most 'feature' RBs. Long scoring plays aren't affected nearly as much, which is why QBs, WRs, (and Chris Johnson) aren't penalized as much.

    I've been working on a much-improved version of the WP model since the summer, and I thought it would be ready by now, but I just don't have the time. It's fairly laborious, and I need large chunks of undisturbed time at home to work on it, which is something I don't have.

    I'd like to wait until it's complete before I charge ahead with more WPA work, but for now this is a fun exercise. At the very least I think it scores QBs well, and like it or not, it's the QBs who are going to be the MVPs.

  27. Anonymous says:

    My solution is to regress the numbers a bit to the mean. For example, in the tennessee game, they had a 0.01 WPA on the last play of the game. Did anyone think that they had only had a 1% chance of winning? It was probably around 10% chance they win.

  28. Jwsinclair says:

    Brian,

    Are you sure there's a bias is the system, or are short scoring plays just overrated in general? Everybody likes to look at how many touchdowns a player scores, but it seems like, if the offensive line does its job, pretty much anybody could score from the 1.

  29. Marver says:

    A system that uses expected points instead of win probability would probably be a hell of a lot more useful/accurate. The fact that it would remove the contribution of the team's defense on how much win probability the offensive players can add makes it worth doing.

    I'm not sure how you can justify that a player who throws for 80 yards and a score on 8 passes in one drive is less of a player than someone else who puts up the identical drive at the identical time in the game, but has a team up by 14 because of a better defense.

    Also, there's a ridiculous amount of play-calling bias clouding these statistics.

  30. Becephalus says:

    I like most of the stuff here, but I didn't find this enlightening at all.

    A fun thing to do with the data, but it tells you almost nothing relevant about the players.

  31. Brian Burke says:

    Marver-I think that's exactly the point of a WPA system. If a team's defense is crushing opponents, then the QB isn't as 'valuable,' as one on a team with a weak defense, is he?

    I'm not sure what is meant by 'ridiculous amount of play-calling bias.' If the coach calls your number, he calls your number and you can either make the play or not.

  32. Marver says:

    Under the WPA system, a slightly above-average QB on a team that puts him in a lot of high leverage situations will grade better than the best QB in the entire league who just happens to be put in a large percentage of unfavorable WPA situations. But by no means is that slightly above-average QB the most valuable in the league.

    My main problem is that it's entirely possible for one player to end a game with a WPA of over 1. If a touchdown drive in a tie game adds .15 (totally arbitrary), yet his defense continually gives up game tying touchdowns, it's entirely possible for a player to end a game with this system saying he was worth MORE than a win that game. I really, really think a points-based system...like the ones you use in the onside kick article, etc...is far more useful.

    Re playcalling:
    You constantly cite game theory to dictate that teams should pass considerably more until they reach a balance in value with respect to run/pass in order to optimize their output. I agree. But you have to apply the results of those articles elsewhere, like here. If a team is too predictable in one area over another, it'll hurt the WPA numbers of those said players even though it's an artifact of play-calling ratios. Adjusting the data for this in order to get a true gauge on player 'value' makes sense.

  33. Brian Burke says:

    Ok. See what you mean on playcalling. That's just the breaks though. Remember that WP and WPA is a narrative of what did happen, playcalling and all.

    Yes, a player could very well end up with > 1.0 WPA in a game, especially if the team around him isn't so good. That's part of being an MVP.

  34. Jeremiah says:

    This was something I had been looking forward to for awhile, but I'm a bit disappointed in how the results turned out. Theoretically the difference between Brett Favre and Aaron Rodgers could be made up in just a handful of well-timed big plays (or poor plays for Favre), which makes me wonder if a 16-game sample just isn't enough to get a good read on these things.

    I guess what I'd like to see is a breakdown of WPA for the players on an individual team. For example, if Peyton Manning has added just 0.8 WPA for an 11-0 Colts team, who is being credited with all of the other wins? The defense?

  35. James says:

    Hey Brian, with the Cowboys players' WPA in the comments, you have "Collie" listed. Is that a mislabled Miles Austin or is that Austin Collie's WPA?

  36. Bob Weber says:

    After tonight's game I would like to see some Cardinals. How do Warner and Fitzgerald fit in?

  37. Anonymous says:

    In a year when the NFL is being dominated 300-400 yards through the air a game, with the #1, currently 12-0 team in the nfl having the league lowest rushing totals (though also tied for the league's 2nd most rushing points), while some of the bottom-feeders like the Raiders seem to do nothing, and lose (except to the steelers o.O), and dothers like the Rams and the Bills try to throw the ball and be aggressive all day get nowhere...

    I think we have ourselves an anomaly =]

    Or...The points here aren't as accurate as they initially seem.
    Oh sure, I could see looking at your avg/rush, distance to 4th down, and 4th down conversion rate and realizing it'd be prudent to go for it a few times you'd have never thought of it before, however...

    This entire article Seems to indicate that Aggression in a game creates scoring and Conservatism limits it on both sides of the ball.

    I would suggest that "aggression" is probably not going to win an underdog a game, in fact, if they are the less talented game, it will probably just make it worse.

    For example, Let's look at the Lions and the Colts. Colts would be a huge favourite, so according to this article, the right thing to do would be for the Lions to play as aggressively as possible, passing deep, avoiding burning the clock, so maybe go no huddle, don't really run the ball if you can help it, except in very short yardage situations, and trust your defense to keep up as you keep them on the field for half a quarter longer than your opponent.

    Further, in order to safeguard themselves, the Colts should play conservatively. Huddling between run plays to run the clock, and not putting pressure on the lions qb, favoring zone coverages instead.

    Ideally, that should work, right? Except, this article completely ignores the dynamic matchups that decide NFL games. The Colts have the #1 passing game with the Last ranked running game, while the Lions are bottom last at defending against the pass, and are in the bottom quarter of the league for passing it themselves, combined with their inability to convert on 3rd down, and an even worse 4th down percentage, and 10 giveaways in 12 games, I really don't think they'd be giving themselves any chance whatsoever to win this game, but the Colts would also be inexplicably toning down what they do best in favour of what they do worst.

    Of course, this is about as an extreme of a contrast as you can get between two teams, but that's why I used it to illustrate the one-dimensionalness of "agression" vs "conservatism" as defining strategies.

    Instead, it's about getting the edge on a mismatch and exploiting it, that's what wins games. So for an upset to occur, the underdog has to find an unexpected exploit, and push it all they can, and hope it's enough. Sometimes, sure, it'll be a hole in their coverage zones that throwing it deep will exploit, or it'll be a bad matchup on the corner with your RE, and you'll play more aggressively on both sides of the ball. Or maybe, it'll be that the Colts' run efense is in the bottom half, and the Lions can chip away at them by running it up the middle. Would it be enough to win? Probably not, but the point is, throwing the game plan away straight away in favour of some ideal is *not* going to work. You have to stick in it as long as you can and keep hoping to find that edge that'll prove you're /not/ the underdog in the matchup, or to realize that you were called the underdog for a reason, and you're going to lose.

    =]

  38. Joe G says:

    You just have to take MVP here to really mean "valuable", not necessarily "better".

    I gave an example a few weeks ago on one of these WP articles about 1 QB leading 5 TD drives and winning 35-10, and another QB also leading 5 TD drives, but winning 35-31. The 2 QBs may be the same skill level, but the second one will rack up WP points because he was always in situations where game is in balance.

    On an expected points scale, they would be equal.

    It's debatable which is better.

  39. Anonymous says:

    I have a quick one. It's almost impossible to measure the WPA of a defensive player... unless he's injured. Can you calculate the average WPA of a play against the Pittsburgh Defense with and without Polamalu ?

  40. Buzz says:

    I would second Jeremiah's comment.

    "I guess what I'd like to see is a breakdown of WPA for the players on an individual team. For example, if Peyton Manning has added just 0.8 WPA for an 11-0 Colts team, who is being credited with all of the other wins? The defense?"

    I'm curious as to how those wins are divied out. I think the defense has done a decent amount this year but the running game hasn't been all that great and special teams are only average at best.

    Also on the .5 or 1. I do think you would need to double all of the WPA's to find out how many games a player "won" for their team. Imagine a 1 play game. The game starts at .5 After the play which we will call an Adrian Peterson run the Vikings either win the game or lose the game but their WPA only moves by .5 As such in this one play Peterson only gains/loses .5 WPA "value" but has clearly either won or loss.

  41. Jim Glass says:

    Two thoughts...

    [] Regarding Adrian Peterson, I've always believed the "great breakaway running back" has tended to be an outright curse for NFL teams. A liability.

    That's because teams that have one often invest their resources in "making the most" of him and the running game, their supposed strength -- diverting those resources from the passing game, which is much more valuable to winning in the NFL. In particular, when a team needs a bunch of yards (say 3rd and long) even the greatest break-away running back is nowhere near as dependable as a decent passing game.

    OJ Simpson's teams averaged 4-10 over his career.

    Gayle Sayers, OJ, Earl Campbell, Eric Dickerson, Barry Sanders -- all had long careers on losing teams.

    Walter Payton in a 13-year career had only one good season on a good team (after Buddy Ryan's 46 defense made the team good).

    Tony Dorsett won a lot, but on a great team with a HOF QB, great OL, top D, etc -- the things it takes to have a top team.

    [] Regarding Marver's comments:

    I think these WPA stats are *great* for a particular purpose -- objectively showing how much impact (plus or minus) a player has actually had in past game situations. We haven't had such a good stat for this purpose before.

    OTOH, this shouldn't be confused with showing "how good" a player is. I think this was Marver's point. The EP stat would be better for this.

    The reason is there's a lot of situational noise, random variation, affecting WPA.

    E.g.: Two players demonstrate the same skill levels, but one happens to break big plays when they matter, the other doesn't, as a matter of chance.

    The WPA method scores the former much the higher -- which it *should*, for its purpose, "who broke the most exciting big plays?" That is, showing past results.

    But the EP method -- or another method that scores performance as per the average situation (best with a strength of opposition adjustment) better shows expected *future* performance, which is what we really mean and expect in the "better player".

    To instead use WPA scores as a measure of ranking best player feeds into all the "clutch performance" fallacies -- the players get ranking bonuses for peforming or failing "in the clutch".

    One field goal kicker is pretty much as good as another, we know. But if during one season one happens to kick an unusually large bunch of long-range, last-second, come-from-behind game-winning field goals, by chance, he'll have a much higher WPA than the others. Should the team's GM give him millions of dollars more against the salary cap to lock him up?

    I'm old enough to remember the Miracle Mets' interstate hitting shortstop Al Weis hit a huge, World Series-changing home run in that miracle series. In baseball terms, Al had a huge WPA in that Series -- but I wouldn't confuse that with him being a good hitter.

    Some while back there was a discussion here about the difference between being "the champion" and being "the best".

    The best has the biggest chance of becoming the champion -- but that's not always how it works out.

    Football is a short season, 16 games. Contrary to the popular saying, the breaks don't always even out (as they mostly do in a baseball season of 162 games).

    As I see it, a WPA metric best looks backward to show how a team actually won a championship (or finished 0-16, or performed above or below expectations) ... while a corresponding EP-type metric would do a better job of measuring quality of play and of looking forward to project future performance.

    If you've got the data, you could publish them both for the same players. The gaps in the numbers would be interesting -- and could set up a whole bunch more of fun "clutch player" "choker" arguments.

    FWIW.

  42. Marver says:

    "I think these WPA stats are *great* for a particular purpose -- objectively showing how much impact (plus or minus) a player has actually had in past game situations. We haven't had such a good stat for this purpose before."

    But I don't even necessarily agree with that. The statistic is a volume statistic, not a rate statistic. If it were all normalized due to opportunity -- like, WPA/(WPA potential gain on said play) -- so we could get a gauge on what the player COULD have done, compared to what he did do, I'd be more inclined to listen to its results. The rest of what you posted clarifies my point, thanks.

    I think a WPA stat like the one mentioned is most interesting if you remove the players and end result from the whole equation. If you only examined the play call and the leverage of the situation, we could understand which teams are relying on which positions more heavily than other teams. I think that, much more than the end result statistic we're given here, would shed a lot more light on the dynamics of this season rather than the randomness that is the result of a few hundred plays. It would also allow us to evaluate which teams aren't properly utilizing their team strengths.

  43. Focus says:

    Troy Polamalu is working with the poor, see what he's doing at: FocusNorthamerica.org/tackle

Leave a Reply

Note: Only a member of this blog may post a comment.