How to Count Unsuccessful Plays for Receivers?

Here's a question. I've been applying the Win Probability model and Expected Points model to individual player performance lately. For QBs, it's fairly straightforward--we add up all the successful events like completions and touchdowns, and subtract out all the unsuccessful events like incompletions and turnovers. But what about receivers? Should we count incompletions and interceptions against them?

Although I have an opinion, I'm honestly not certain, and I'd appreciate everyone's input. One one hand, if we only count positive plays for the receiver they're going to have very large WPA and EPA totals out of proportion with those of other positions. On the other hand, if we do count incompletions and turnovers we'll be penalizing receivers in many cases when they had little or no control over the outcome of the play.

Before you form an opinion, think about what we're trying to measure, or at least think about what we can realistically measure given the play-by-play. A stat should be designed to try to measure a certain attribute or answer a certain question. The question ideally should come first, and the metric should come second.


I'll wait to hear other opinions before I share my own, mostly because I'm open to suggestions. But the other reason is the psychology of blog comments. If I said we shouldn't count negative plays against receivers, I'd get 10 disagreeing comments for every one that agrees. And if I said the opposite, I'd still get 10 times more disagreeing comments than agreeing.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

31 Responses to “How to Count Unsuccessful Plays for Receivers?”

  1. Alchemist says:

    While I love what you do on this blog, and I'm intrigued by your application of WPA to individual players, I think it starts to become very sketchy once you go beyond the QB and RB. This happens because your play-by-play narratives are quite limited. Despite this sparseness, I think the evaluation is still useful when looking at QBs and RBs because so much of what they do is still embodied in your play-by-play stats. But I would argue that for WRs (and for all other positions, for that matter), so much of what they do is NOT embodied in your play-by-play narratives.

    When a WR is open but is not thrown to, this looks on the play-by-play as a non-play. But of course, the WR was contributing as much as he could.

    When a WR is given close double coverage and is not thrown the ball, he has contributed handsomely to the play by drawing 2 defenders away from the action. But this looks on the play-by-play to be a non-play for the WR.

    When a WR throws a key block, this is not reflected on the play-by-play.

    I'm not sure if your play-by-play narrates targets (e.g., "Manning throws incomplete to Wayne"). But even if it does, that narrative is probably woefully incomplete. Did Manning throw a bad pass? Was Wayne even open? Did Wayne drop the ball? Did Wayne run the correct route?

    And as for interceptions, deciding which ones should be charged against which receivers (if they are charged against the receivers at all) is extremely subjective. I've seen 4-INT days that were almost completely the fault of the QB, and I've seen 4-INT days that were almost entirely the fault of the receivers.

    So long story short, I'm skeptical that this stat will truly be illustrative. The question I assume you're trying to answer is, "How much impact did this receiver's play have on the outcome of the game?" and I'm not sure that any WPA analysis, based upon your current play-by-play stats, would come close to answering that question.

  2. John says:

    To try to answer your specific question, I don't think it's any more or less messy than what is already being attempted with quarterbacks and running backs. Quarterbacks are just as sensitive to bad hands and bad routes as receivers are to bad throws, throw aways, tips and desperation throws. Running backs, likewise, are sensitive to line play and the overall threat of their respective team's passing offense.

    Nevertheless I would argue that it's better to include all plays including incompletions and interceptions. It won't be accurate, it won't be less accurate than just including only receptions, but it will tell us the whole muddy picture. It will tell us, instead of how good the receiver was, which is impossible with current stats, how valuable passes targeting that receiver were. That's the best we can hope for.

  3. JMM says:

    It's a team game. If a player was on the field for the play, he contributed to the success/failure of the play.

    As long as the name and description indicate it is a high level measurement, not an indication of ability to catch, it should be fine and be an addition to the basket of stats.

  4. Alex says:

    Following on what John said, you're already counting incompletions against QBs even though you don't know how much of that was the receiver's fault. So if you feel that was an easy decision to make, it should be just as easy to count incompletions against the receiver. The only potential problem I see here is that you'll potentially double-count passing plays (both good and bad) if you try to work from individual assessments to saying something about a team.

  5. Alex (#2) says:

    Wholeheartedly endorse John's comment

  6. Big Jim says:

    I also endorse John's position. If we believe that
    Data -> Information -> Knowledge -> Wisdom, then
    it seems to me job #1 is to include all the pertinent data you can get.

  7. Aron says:

    I'd like to see events weighted inversely with WPA. Short completions after a game is all but decided come easily. For example, why credit Seneca Wallace and crew for their barrage of completions against the Colts when they mostly came after the Colts put the game out of reach?

  8. ASG says:

    I think only drops should count on the WRs. If a ball is intercepted, there are few times the WR could have defended the past but most of the time the WR was covered as much as any other WR on the field or was open and the QB made an errant throw, either by throwing the pass poorly or throwing it at all.

  9. Jeremiah says:

    I would go with all plays. I haven't done any systematic research, but it does seem like Catch Rate is fairly consistent for most WRs, even as they change teams (eg, Chris Chambers always having low Catch Rates). I also have done some preliminary research showing a correlation between college Catch Rate and NFL success, which further backs up the idea.

  10. Bob Weber says:

    I agree with the 'include all data' position. The more the better. In fact, I would argue that a throwaway incompletion should be counted against all the receivers on the field during the play.

    Like someone said above, a WR value is more than just catches and yardage. Does that player consistently draw double coverage allowing other receivers more opportunities? Should it be counted against the receiving core when the QB has to scramble, or just get rid of the ball?

  11. Anonymous says:

    I think this discussion illustrates a major flaw in football play-tracking statistics: the separation of credit on passing plays. I think there are two major components that are not adequately tracked.
    1) Incompletions and Interceptions: When a QB clearly under/overthrows a receiver or makes some other error, can we penalize the receiver for being involved in the play? How can we determine from game logs if he was out of position? Should a QB be penalized when a ball hits a receiver in the hands/chest and then bounces incomplete or is intercepted? In the absence of baseball-like error scoring I think both QB and receivers (whether WR, TE, or RB) have skewed statistics, but given the number of plays each is (directly) involved in, these biases will have much more effect on the measurement of receiver contributions.
    2) Yards/Completion: As more of a general complaint about the way we track passing plays, there is no separation of yards before/after the catch. When a QB drops a ball into the arms of a receiver 40 yards downfield, it is probably the skill of the QB that contributed to the bulk of the play's result. However, when the QB throws a 5 yard pass over the middle and the receiver picks up 35 yards by making defenders miss tackles, it is probably the skill of the receiver that contributed to the bulk of the play's result. I don't think the two results are actually in balance, though I don't have the data to support that assertion.

  12. inztant says:

    I see WPA as an advantage added to the team, leading me to look at these types of things:

    (20+ yd plays)/total times thrown to

    times thrown to/total team passes

    Maybe there are others. I am not a fan of redzone stats, but times thrown to in the redzone should reflect reliability in big situations, ie holmes was thrown too twice in a row in pits winning drive in SB43, althought this could weight heavily towards TEs.

    Brian, I would like you to do a little experiment. See what kind of win percentage you get using ONLY offensive stats or ONLY defensive stats. I am trying to do some basic stuff using efficeincy numbers to predict points by optimizing a coefficient behind each: C1*OPss+C2*PRsh-C3*DPss-C4*DRsh, and BOTH defensive coeff ALWAYS are 0.

    I realize that using eff stats to predict points may not be the smartest idea, but you would think the opponents defensive eff stats would have at least a SMALL say in the amount of points scored... I may be doing something fundamentally wrong here, as this is the first time I have ever tried anything besides just having a rating for each team, so any input is greatly appreciated. Love your blog btw...

  13. Lab says:

    I think the answer is to track the data both ways, and then compare the results. Either approach can lead to mis-evaluation of some receivers, but if a particular WR is solid (or weak) under both approaches, those consistent results will carry more weight. And for those WRs with inconsistent results, the very nature of the inconsistency might lead to some interesting analysis.

    Even better would be to track 3-4 different approaches, and look at what the consistencies/inconsistencies tell us about the WRs.

  14. Anonymous says:

    if you have the data, use only plays where receivers touched the ball and the play was still an incomplete to penalize them

  15. James says:

    I say the more information the better, but if you want to cut out bad plays you absolutely have to count drops. Then again I imagine there's a disrepency between games as to what's a drop and what's not.

    The precedent set by the QB system says you have to include all data. Another way to look at it would be to break up "Air yards" and YAC and assign air yards only to the QBs and YAC to the receivers. Of course I see the problem here (RB dump/screen passes get a lot of credit, double-teamed WRs, etc), but what else can you do? A 75/25 split of each stat?

  16. Huw says:

    Maybe you could factor in the receiver's historic incompletion or interception rate vs incompletion or interception rate for that position on that team as a whole as a way of estimating the proportion of blame (and so WPA and EPA loss) for a specific play?

    Ditto for the yardage of the play, whether it's a first down or touchdown, etc -- you could use historic data to estimate the likely contribution of the QB and receiver. If you want to get really crazy you can compare how a receiver did with a change of QB -- something like the platooning data used in baseball analysis.

    To use the Eagles as an example, McNabb has always been a productive QB, but his output was much higher in years where he had a big-play receiver -- TO or DeSean Jackon. Did he become a much better QB in those years? Not really -- presumably his stats when throwing to other receivers were fairly steady. So we would give Jackson and Owens a higher proportion of the WPA and EPA than, say, Hank Baskett. Owens and Baskett's careers with other teams can give us an idea of the sort of boost they give a QB's stats.

  17. Nat says:

    Counting receivers individually would be doubling the actual success on a play. On a successful pass play, that success is brought out in the QB's statistics, but if you count the success for both the QB and the WR, you count the success twice. The quarterback's stats also somewhat represent the receivers, because drops count as incompletions and so forth.

  18. inztant says:

    ANOTHER QUESTION:

    what program do you use to do all of these calculations?

  19. Anonymous says:

    A receiver job is three fold, to get open, to catch the pass, and to run with it. Since you can't judge whether the receiver was open by the play-by-play (unless you use intended for), the open stats we got are catches, intended for (and drops), and YAC. any positive must have a negative catches and drops, and YAC and no YAC. That's how I would judge recivers

  20. Anonymous says:

    inztant:

    Ignore my question about my formula, I screwed up.

  21. Anonymous says:

    It depends upon what you want this statistic for. Is it one of the many useless statistics I see on other sites? Or is it to be a useful one, allowing the user to compare performance against other receivers on the same team? Or4 across the entire league?

    Maybe none of what you propose is adequate! Maybe you need to count how many times the reveiver got open, he did hi sjob.

    Ron Jaworski did an analaysis of Cutlers interceptions. Of teh 26 he threw this year, I think he said 2 were the receivers fault. Maybe 2 were bad luck, incredible defensive plays. And teh vast bulk Cutlers fault.

    So why penalize receivers for something that is 80 - 85% the quarterbacks (or defenses) fault.


    As a Jets fan, I saw Sanchez make some awful throws for incompletes, no way they should penalize the receiver. And most of his interceptions were his fault.

    Maybe a quarterbacks receivers don't do a good job of getting open and the QB takes a risk. Then its indirectly more the receivers fault.

    Maybe the receiver is on a weak team and his catches dont add much to the WPA because of the score.

    Or maybe the receiver is on a strong team and already up a touchdown so the WPA isnt taht high.

    In short I am not convinced this is going to be a useful stat.

    Perhaps you could redirect your effort into an offense based stat like "How often did 1 or more receivers get open, giving the QB a realistic chance at a completion". Mmeasur ethis bothe ways, for the defense as well.
    The hard part is getting this data.


    - Jets fan

  22. Anonymous says:

    Doesn't play-by-play specify where the pass was thrown? (i.e., pass short left, or pass deep right) If so, I think you should discredit WRs based on where the pass went. Say 80% of short passes are complete, then 80% of the negative WPA goes to the WR. If 40% of long passes are complete, then an incomplete pass deep is only 40% to the WR.

    I don't know how to count INT's though. Maybe half of the completion percentage (40% and 20% above)? Or you could switch those two numbers--the completion percentage (80% and 40%) count for INTs, and incompletions are half of that.

  23. coldbikemessenger says:

    I would count all incompletions against all recievers on a weighted basis. I would not count interceptions against recievers

  24. coldbikemessenger says:

    I mean there are something like 200-225 incomplete passes a year right? add up all the negative points for those and then come up with a formula to share the blame.

  25. Andrew Foland says:

    The question I propose to answer: how much more (or less) did this receiver contribute to winning, than an average receiver would have in the same set of situations?

    The method: set up a set of "equivalency classes" a la Krasker. They probably don't have to be as strictly equivalent as Krasker's construction. (And I've left kind of vague exactly how to do it; that's the real work that makes a guy worth being picked up by the NYT, which would be Brian, not me!) Find average WPA over the league for each category. For each receiver mentioned in each play, add WPA for the play, subtract average WPA over the league for the equivalency class, summing over all plays for the receiver.

    This sort of statistic ought to cancel out a lot of the "average" biases for when receivers get mentioned or not, and whether it's the QB or not. Of course, receivers with truly awful or extraordinary QB's will still suffer, but I suspect much less so than a straight WPA sum.

  26. Anonymous says:

    Drops are a good negative stat.

    Actually, for wide receivers you could set up a "par" (stats par) for their position.

    You could then measure them against the "par"

    This does not solve your problem with EPA and WPA calculations, but it would be a good way to measure their performance.

  27. Jeff says:

    I don't really have an answer to this question, but I've long been an advocate of not calculating INT or fumbles, but rather just have a category called "turnovers" under which you can assign fault more readily. Obviously there would be a subjective element added that we don't have now, but the bright-line rule we have now always penalizes QBs for both INTs and fumbles on exchanges, when the reality is they are often not at fault.

  28. Brian Burke says:

    Great comments. Thanks to all. I'm drafting my own thoughts now, which agree with most of the comments here.

    Iztant-I use a combination of things. For regression and statistical stuff I use GRETL. For most things I use Excel, and pivot tables in particular. A lot of what I've been doing lately is coded using my own PHP scripts.

  29. Anonymous says:

    I haven't read all the comments, so maybe this point has been made. But to temper the effect a bad QB can have on a talented WR, you might include the teams combined QB passer rating as a data point for the WR.

  30. Anonymous says:

    Dropped passes are about the only negative stat you can fairly give a receiver. The criteria for a dropped pass imo would be a pass that a receiver touches with both hands and at least one foot on the ground or while running and does not catch.

    You can make an argument for a tipped balls resulting in ints stat but imo that would be going against the fundamentals of how passing stats are already recorded which is: Point A of the ball (the passer)and point B (receiver, defender, or incomplete) - adding a stat for a deflection would create a point C. It would also diminish ints accredited to defenders, a tipped ball or otherwise broken up pass only becomes an int because a defender made a good play - being in the right position, tracking the deflected ball, and finally making the catch. This is going above and beyond what is expected of a defender and is the most prestigious stat for DBs and Safeties, this should not be compromised by accrediting the WR with contributing to an int, they should only be accredited with making or not making the catch, after that it's up to physics and the defender's ability to decide if it becomes an int.

    An offensive deflected pass stat may be another option to consider for logging a receiver that "becomes a defender" on a play where a bad pass or deflection is involved

  31. JetsNeedNewton says:

    Jets fan hit the nail "Perhaps you could redirect your effort into an offense based stat like "How often did 1 or more receivers get open, giving the QB a realistic chance at a completion". Mmeasur ethis bothe ways, for the defense as well.
    The hard part is getting this data." I was googling this stat when I ended up here. This is a critical stat because for the Jets, the question is: Is Sanchez a bad QB or are his receivers never able to get open?
    Recievers seem to be measured by thier acrobatic one handed catches in critical moments. But the key to how good they are is how many times they do thier jobs and get open, providing the opportunity for success.
    The one thing you would have to add is time to compenstate for other factors like a weak front line. That would vary from team to team. So in the Jets case, the measure would be: "Does the reciever get free within 3 seconds" because in the fourth second the QB is either running or getting crushed.
    The data will be hard to get because unless you have feet on the ground at the game focused on counting, TV coverage may not show you all recievers. I'm going to give it a try using NFL rewind to count open recievers in the Jets\Giants debacle. If anyone cares, I can come back and list the stats.

Leave a Reply

Note: Only a member of this blog may post a comment.