Analyzing Replay Challenges

The new WP model allows some nifty new applications. One of the more notable improvements is the consideration of timeouts. That, together with enhanced accuracy and precision allow us to analyze replay challenge decisions. Here at AFA, we've tinkered with replay analysis before, and we've estimated the implicit value of a timeout based on how and when coaches challenge plays. But without a way to directly measure the value of a timeout the analysis was only an exercise.

Most challenges are now replay assistant challenges--the automatic reviews for all scores and turnovers, plus particular plays inside two minutes of each half. Still, there are plenty of opportunities for coaches to challenge a call each week.

The cost of a challenge is two-fold. First, the coach (probably) loses one of his two challenges for the game. (He can recover one if he wins both challenges in a game.) Second, an unsuccessful challenge results in a charged timeout. The value of the first cost would be very hard to estimate, but thankfully the event that a coach runs out of challenges AND needs to use a third is exceptionally rare. I can't find even a single example since the automatic replay rules went into effect.

So I'm going to set that consideration aside for now. In the future, I may try to put a value on it, particularly if a coach had already used one challenge. But even then it would be very small and would diminish to zero as the game progresses toward its final 2 minutes. In any case, all the coaches challenges from this week were first challenges, and none represented the final team timeout, so we're in safe waters for now.

Every replay situation is unique. We can't quantify the probability that a particular play will be overturned statistically, but we can determine the breakeven probability of success for a challenge to be worthwhile for any situation. If a coach believes the chance of overturning the call is above the breakeven level, he should challenge. Below the breakeven level, he should hold onto his red flag.


To calculate the breakeven probability of reversal, we need a bit of algebra. Let's define the relevant numbers as follows:

B = breakeven probability of reversal
R = win probability given call reversed
U = win probability given call upheld
N = win probability given no challenge is made

We just need set the WP of no challenge (N) equal to the "lottery" of challenging the call and we get:

N = B * R + (1-B) * U

Solving for B, we get:

B = (N - U) / (R - U)

This makes sense, because the lower the WP penalty is for a failed challenge, the lower the breakeven success probability needs to be. Likewise, the bigger the WP bonus is for reversal, the lower the breakeven success probability needs to be.

Let's look at the challenges from Sunday's games.

1. 1st-10-ATL 16, 4:34 in the 1st, NO up by 6 -- NO challenges an ATL 17-yard pass completion. The play stands as called. The Saints' WP for no challenge (N) is 0.373, which is a 1st-10 for ATL at their own 33. A successful challenge would give ATL a 2nd-10 from their own 16. The WP for a successful challenge (R) would be 0.686. The WP for a failed challenge (U) is the result of the play minus a timeout for NO, which is 0.619 . This makes the breakeven...

B = (.627 - .619) / (.686 - .619)
B = 0.12

So as long as Sean Payton thought he had better than a 12% chance at overturning the call, it was a wise decision to challenge the play.

2. 2nd-6-CHI 18, 0:15 in the 3rd, tied -- A CHI pass is called incomplete. CHI challenges the call and it's overturned, resulting in a 7-yard gain and a 1st down. I'll skip the math from here on out--the breakeven probability was 10%.

3. 3rd-3-WAS 31, 3:21 in the 1st, scoreless -- WAS's deep pass ruled incomplete. WAS challenges, hoping for a nearly 50-yard gain. The call was upheld and WAS charged with a timeout. Breakeven was 5%.

4. 2nd-2-HOU 22, 6:41 in the 2nd, still scoreless -- A WAS run gains 21 yards to the HOU 1, but the ball squirts out after the tackle. HOU challenges, hoping for a turnover, but the call is upheld as WAS kept the ball. Breakeven was 3%.

5. 3rd-8-PHI 46, 3:27 in the 4th, tied -- JAX's pass appears to be caught then fumbled to a PHI defender. The call on the field was an incomplete pass. PHI challenges but the call is upheld. Breakeven was 5%.

6. 1-10-CAR 41, 14:57 in the 3rd, CAR ahead by 10 -- TB's Josh McCown escapes a sack and completes a 4-yard pass. CAR challenges, hoping McCown was down before throwing but the call is upheld. I commented that I believed it was an unwise challenge at the time, but the breakeven was only 7%.

These six examples are instructive. First, contrary to my intuition, it seems that coaches are not blowing their timeouts on frivolous challenges. If a call is fishy at all--say somewhere in the neighborhood of a 1-in-10 or 1-in-20 chance of being overturned, and the difference in outcomes moves the WP needle at all, it's probably a good idea to challenge it. Certainly any time a score or turnover is on the line, it should be challenged. When a conversion is at stake, or even when the probability of a conversion is significantly affected, it's probably a good idea to challenge.

Things change late in close games when timeouts begin to skyrocket in value. But until then, there are just too many different and more important factors than timeouts. A team could end up ahead, and not need their timeouts at all. Or they could easily end up where they would lose even if they had six timeouts.

The next step is to generalize this approach to produce a model of when to challenge a play based on score, time, and play situation. And even better, perhaps I can make a set of rules-of-thumb that can be simple enough to use on the sideline.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

5 Responses to “Analyzing Replay Challenges”

  1. mmcnair says:

    At first I thought you had an error in the ATL/NO example, but I think it's just a problem with the perspective. I'm pretty sure the .373 is from NO perspective but the rest of the numbers are from ATL perspective.

    The B = 12% seems to be correct if I just substitute in (1 - .373) = .627. So it should read:
    B = (.627 - .619) / (.686 - .619)
    B = 0.12

  2. Anonymous says:

    In the Saints/Falcons game, NO was leading 6-0, not "down by 6"...

  3. Brian Burke says:

    Oops. Yep.

  4. bones says:

    Play #2 is, interestingly, slightly more complex. CHI challenged and it could have been ruled a catch, followed by a fumble recovered for a five-yard gain (short of the first down). So, the breakeven probability numerator would be even a little bit lower, by the probability of the catch+fumble scenario times the difference in WP between the catch+fumble and an upheld call.

  5. Brian Burke says:

    -Fixed.

    -Thanks, Bones. Important point that's not in the pbp.

Leave a Reply

Note: Only a member of this blog may post a comment.