Rethinking the Massey-Thaler Draft Study

Economists Richard Thaler and Cade Massey authored a widely-read research paper analyzing the value of NFL draft picks, and they've recently published an updated version. The paper's primary finding was that teams are overconfident in their ability to choose the best players. In essence, the very top picks are overvalued relative to later picks, both in terms of what teams are willing to trade to move up in the draft and in terms of salary.

Recently Richard Thaler penned an article for the NYT discussing his paper. But puzzlingly, he goes on to make a claim that clearly contradicts his own research. He writes, "it makes absolutely no sense to be giving so much money to unproven rookies, many of whom turn out to be busts." Further, he writes, "veteran players would probably agree with the principle that eight-figure salaries should be reserved for players who have already proved themselves on the field."

The Massey-Thaler paper is very sound and persuasive, and perhaps the most concrete finding is that draft picks in all rounds are underpaid compared to free-agent veterans. In short, the study assigns a value to a player based on a variety of measures such as Pro Bowl selections, games started, and years on a roster. The research finds draft picks are paid less than free-agent veterans for the same expected performance.

This graph taken from the paper is the study's bottom line. The black dotted 'compensation' line is the average annual pay for each draft pick. The green 'performance' line is the salary a team would have to pay a 6-year veteran free agent for the same expected performance. The blue dashed 'surplus' line is the difference between the two pay levels.

Surplus value peaks shallowly at the bottom of the first round and through the second round. That's where teams get the biggest bang for the buck. But still, the draft pick surplus is significantly positive throughout the entire draft. Contrary to Thaler's remarks in his recent NYT article, draft picks are bargains compared to veterans.

According to Massey and Thaler, although the top picks appear to be overpaid relative to later picks, draft picks in all rounds are underpaid relative to veteran players. Thaler's own research contradicts his recent statement that it makes "absolutely no sense to be giving so much money to unproven rookies." It makes perfect sense to me.

Now that I'm thinking about the Massey-Thaler paper again, I'm wondering if the overvaluation of top picks might be justified, at least to some degree. Massey and Thaler seem to assume a linear "bricklayer" model of football player production. By that I mean that a bricklayer who lays 10 bricks per minute and makes $10 per hour can be replaced by two slower bricklayers who each lay 5 bricks per minute and make $5 per hour. In this model, absolute performance is what's important.

But one great QB cannot be replaced on the field by two average QBs, and a team can only field 11 players on the field at a time. It can only carry 53 players on its roster. The bricklayer model does not apply. Further, relative ability is what matters in football, not absolute ability.

As I wrote last year, football players are more like gladiators than bricklayers. When two gladiators square-off in the Coliseum, and one is slightly but consistently better than the other, the superior gladiator would almost always win. The winner survives, basking in glory, while the outcome isn't so pleasant for the loser. The NFL is similar to gladiatorial combat: Every game is a winner-take-all competition, and there can only be one champion. Put simply, if you're not among the very best it matters little how good you are in absolute terms, which is what the Massey-Thaler paper measures. The nature of football puts a premium on singular excellence, and that's why the players with the highest likelihood of great success are prized so highly.

While I trust the statistical findings of the paper, it doesn't suggest that rookies are overpaid. It suggests the opposite. And further, there may be a good reason why the very top picks are so highly valued. It may not be due to the overconfidence and irrationality of team executives.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

17 Responses to “Rethinking the Massey-Thaler Draft Study”

  1. Phil Birnbaum says:

    What measure did Massey and Thaler use for measuring draft pick quality? If it's "value over replacement level," that probably isn't a problem. If it's value above average, or value above star, or value above a player who doesn't play at all ... then it's not going to work.

    Imagine that you can pick up a free agent or undrafted quarterback for $0, and, with an average team otherwise, you'll go 4-12. The value of a different QB is how many games you'll win *over and above* the 4 wins the no-name guy gives. So a QB who will get you to 12-4 is twice as valuable as they guy who will get you to 8-8, because he adds 8 wins instead of just 4.

    If Massey/Thaler did it that way, it should work. If they did it another way, it won't work.

    If they assumed quality is based on raw yards, then, yes, the great quarterback is probably undervalued.

    Anyway, I guess I shouldn't explain this to you, you know all this already. So, what did Massey/Thaler do to convert QB performance into dollars?

  2. Brian Burke says:

    They didn't use yards or points. To be able to compare player performance across positions, they used Pro Bowl selections, years as starter, years in league...stuff like that.

    To evaluate the value of a draft pick, they estimated the salary that a 6-yr veteran free agent would command given the same expected performance.

    I don't believe they used any kind of 'above replacement' adjustment, and that's something I hadn't thought of. I'm not sure it would make a difference though. In the NFL, I would think a team full of undrafted free agents would go 0-16.

    But I guess it's not that simple. Football player performance isn't linearly additive the way it (almost) is in baseball. There are all kinds of interaction effects. A replacement-level RB might do just fine behind a solid offensive line, but he'd be dead meat behind a poor line.

  3. Brad says:

    Without going in-depth and actually -- you know -- reading the study, I've got a question here.

    How do they rate "expected performance" for an untested rookie? I think expected performance for a veteran is relatively understood, because you've seen their output on the field and you know what they're capable of. A rookie, though, you're taking a big chance.

    How much does "possibility of a bust" factor into the analysis? Because as you point out, you can only hold 53 players on your roster. The difference in pay might be an understandable premium for taking proven performance over an unproven rookie. The opportunity cost of a rookie is not just the money lost if he doesn't pan out -- it's also the opportunity cost of wasting a roster spot over a proven veteran that has a higher probability of actually being a contributor on the field.

    Granted, I'm sure they somehow are controlling the expected performance of drafted rookies, but I'm not sure how well they control "expected performance" beyond average expected, to include variance, and how they control average & variance for veterans.

    If you can summarize from the actual paper, it might shed more light on this.

  4. Phil Birnbaum says:

    Right, I remember now. That seems a decent way of doing it, if they just looked at the market value for that level of performance.

    But I also remember (as you say) that they used number of games as a starter, and whether the guy was a pro-bowler -- but nothing in between. That's a very rough measure of performance.

  5. Alchemist says:

    Full disclosure: I haven't read the paper and I'm only going off your synopsis.

    While I am always interested in these kinds of studies and I'm sure that the paper you reference is of high quality, these analyses typically suffer from a common flaw - how to measure the value of draft picks (especially when you are trying to measure the value of ALL picks across ALL positions). Your comments above indicate that they have fallen back on the "standard" conventions - Pro Bowl selections, years as starter, years in league, etc.

    The problem here is that every team I've ever seen suffers from a massive case of "sunk cost fallacy". In fact, you wrote an article about this last year regarding Jamarcus Russell, so I'll use him to illustrate my point.

    While Russell isn't going to the Pro Bowl any time soon, he has already racked up far more starts than is probably justified by his talent level. I contend that if Russell had somehow been a 7th-round choice, he would have been lucky to ever see the field once an NFL coach saw his actual performance in training camp. But of course, he was a 1st-round choice, which means that he is basically given every possible chance to justify his draft position. He receives more starts and more seasons than he really should.

    Thus, if we are using starts/seasons as measuring sticks, the "value" of a high draft pick becomes a self-fulfilling prophecy.

  6. JKL says:

    I'll re-post something I wrote as a comment a few years ago about the Massey Thaler study, as it applies to your gladiator-bricklayer analogy.

    "I am not confident that the early 2nds are actually more valuable than the early 1sts. I think there are two problems that they are not fully accounting for:

    1. QB's. Many early drafted QB's do not start 8+ games right away, and it doesn't necessarily mean they are busts. Most QB's drafted in the first rounds are drafted early, at the point where the M-T curve is lowest in the first round. Are they missing something about how teams value QB play, and how teams are willing to sacrifice year 1 for greater benefits in later years?

    2. Their crude categories. They go from 8+ starts, to pro bowl, with nothing in between. Thus, Philip Rivers and Charlie Frye are equals in their methodology. If early first rounders who are 8+ games starters but not pro bowlers tend to be in the upper 25% of their position more than others, then their methodology would undervalue early picks.

    They try to address that issue by saying that among the stat skill positions, the production per dollar is higher at the end of the first and early second than in the early first. Fantasy drafters who use VBD should recognize the potential flaw here. Not all yards are created equal. The difference from 800 yards receiving to 1200 yards receiving is worth a heck of a lot more than the difference from 400 to 800. Their arguments do not convince me that they are not undervaluing the higher chance of upper level (just below pro bowl) performance."

  7. Anonymous says:

    To follow up on Brad's post, it seems the only way to justify "rookies are overpaid" would be risk aversion. NFL teams are risk averse and will will pay higher for the same expected value from a lower risk investment. However, I don't know that veterans are necessarily lower risk. Was that in the study?

  8. Anonymous says:

    whoa. I hit a full stop right here " the same expected performance."

    sure, draft picks may be cheap compared to veterans based on " the same expected performance" BUT in terms of actual performance, i would expect veterans to be about twice as reliable as a draft pick (based on a rough rule of thumb that half of draft picks bust, whereas you already know your veteran has performed at nfl levels).

  9. Kulko says:

    @ Anon
    Well you forget the second part oif the equation:
    If you can expect the same, but you have the risk of a bust then you also need to have a considerable risk of a boom. And as several people pointed out, for most mediocre teams its booms in performance that make the difference between reaching the playoffs or not.

    Also draft picks do usually have upward potential in performance inlater years, while for veterans its mostly performing now or never.

    So if you want to spend a rooster spot on a player who is not quite there yet, a rookie is a much better deal then the veteran and it costs more.

  10. Ben says:

    So high first round picks are worth more than veterans, for the money, and mid to high picks are worth even more. So his point that you should be paying more money to veterans is obviously off-base, as rather you should be paying more to mid and high picks.

  11. ch says:

    Does anyone really think that comparing what the top pick made in 1995 is valid in today's draft market? Ki-Jana Carter got a contract for $26.7 million in 2009 dollars, and that was for 7 years. Stafford's contract was for 41.7 million and one less year. I would be very, very surprised if there was still surplus value if you were using contracts from 2005-2009, not 1994-2009. The huge Stafford and Russell type contracts are being averaged out with contracts that simply aren't possible in today's market.

    Another serious problem with comparing rookie and veteran contracts is survivorship bias. To get a veteran FA contract you've got to be pretty good since the average NFL career length is less then the length of the rookie contract. That's going to inflate the cost of the veteran contract since team's are getting a known quantity that's much easier to measure then a draft pick.

  12. Brian Burke says:

    ch--Salaries have certainly gone up over time, but the paper takes that into account.

    Also, while it's true most people put a premium on certainty, it's not purely rational. The draft pick may have a bigger risk of being a bust than a FA, but he also has a bigger upside potential of being a superstar.

  13. Anonymous says:

    I haven't read the study, but I'm going to assume that draft picks don't get overpaid. I think the misconception would be the top guy getting paid substantially more than someone from the middle of the first round. In professional sports, a handful of athletes are paid a ton, while most of them get a lot less, but we only pay attention to those who make a ton. So even if the third overall pick gets a lot less than the first overall pick, we pay attention to the first overall pick, especially when he ends up being a bust.

  14. ch says:

    If the paper takes salary growth into account the graph above sure doesn't reflect that. A 4.5 million / 6 year contract (what the graph suggests for the #1 pick) would be slightly more then what Ki-Jana Carter got. The Lions are paying just a tad bit more. Assuming you believe their 'performance curve' a 6 million dollar average would be above that line and represent negative value. And that's just what Stafford's guaranteed to make, if he made all of his incentives it's a 12 million dollar per year contract - more then Drew Brees makes.

    Teams understand that the top picks are wildly overpaid relative to their performance, and nobody wants to trade into the top 4 anymore as a result.

  15. Anonymous says:

    Garbage In = Garbage Out.

    Pro Bowl selections is probably the worst possible way to measure true success in the NFL.

    In 2009 the Quarterback selections for the AFC were Peyton Manning, Tom Brady, and Phillip Rivers. When Brady and Rivers feigned injury and Manning sat for the Super Bowl, Matt Schaub, Vince Young, and David Garrard were selected.

    According to the 2009 WPA numbers, the top AFC Quarterbacks were... (list only long enough to include all selections)


    There were three better QB's than Brady. Schaub got in as an alternate, but Roethlisberger and Palmer were passed over for Vince Young, and Orton, Flacco, Gradkowski, and Henne were passed over for David Garrard.

    On the NFC side the best 3 QB's were voted in, but when Favre and Brees sat the 4th best QB, Eli Manning, was passed over for Donovan McNabb and Tony Romo... and while the only available QB left in the NFC better than Romo was Manning, both Kurt Warner and Matt Ryan posted better WPA numbers than McNabb.

    This happens at every single position in football.

    Grading player success on Pro Bowl appearances is like determining Grade Point Averages based on votes for the Homecoming Court.

    It's idiotic.

    Why go through all the trouble of deriving a new statistic that more accurately measures how successful a player is at his position... and then using a popularity contest for anything more than a mockumentary?

  16. Aaron from Teaneck, NJ says:

    The real issue is relative value & ROI. Its very non-parametric. Each team has a set of position based values on their 53 man roster. There is no way to model each choice based on a stable equal weighting by team. There are 32 teams with different needs, values they place on each position & weightings that are placed on each need by the staff for each team. There is no way to standardize it.

  17. Anonymous says:

    I'd be curious to see a study of anticipated value of free agents against their actual performance after changing teams. I haven't done any research, but anecdotally, there seem to be an aweful lot of busts in the FA market too(Jason David, Adelious Thomas, and Albert Haynesworth come immediately to mind). In fact, I'm struggling to recall FAs that made an impact with their new team commensurate with expectations (Reggie White, Randy Moss...???)

Leave a Reply