First downs are easier to analyze because they almost always begin with 10 yards to go. Unfortunately, 2nd downs aren't so cooperative. It's amazing how thin he data gets sliced up. Most downs aren't losses, even fewer have holding penalties, and rarely are they declined. Still, there are enough cases for a solid analysis using 1st-down conversion probability as the bottom line.
Put simply, a defense would prefer to decline a penalty on a 2nd down play whenever the resulting 3rd down situation leads to a conversion less often than the 2nd down plus the 10 yards.
The chart below plots conversion probability for 2nd and 3rd down situations. The red line illustrates the conversion probability of 3rd down and X to go situations. For example, 3rd down and 7 situations are converted about 40% of the time.
The green line illustrates 2nd down situations, but slightly differently. It plots conversion probabilities for 2nd down and X plus 10 yards. For example, 2nd and 13 (i.e. 3 + 10 yds) situations are converted 45% of the the time. The black line is the smoothed line fitted to the 3rd down conversion rates. I plotted things this way because it's the actual comparison we're interested in, given a gain of zero yards.
We can see that the conversion probabilities of a 3rd and 7 and a 2nd and 15 are roughly equal--about 40%. That means that a defense should be indifferent to accepting or declining a penalty following a loss of 2 yards on 2nd down.
We can repeat this process for all 2nd down distances to go, establishing a break-even point of indifference for loss amounts. On 2nd and 1, a defense would want a loss of 4 or more yards before declining a 10-yard penalty. On 2nd and 3, they would want a loss of 3 or more yards. And so on.
Reading the graph above isn't going to be practical for a coach on the sidelines. So I thought back to my flying days and thought a rule of thumb would be handy. When doing math in a F-18 (you'd be amazed how often you have to--how much gas will I have left when I get back to the carrier?), it pays to simplify things because there aren't that many brain cells left over after all the flying stuff is worried about. Rules of thumb may be inexact, but the trade-off between precision and speed is well worth it.
In this case, the 'loss of indifference' yardage fits neatly to the following:
4 - 1/3 * original distance to go
For example, from a 2nd and 6, a defense would want (4 - 6/3 =) a 2 yard loss or greater before it declined a 10-yard penalty.
How well do coaches approximate this rule of thumb? Surprisingly well, just like they do for 1st down penalties.
The plot below shows the rule of thumb in red and the actual loss of indifference for coaches in blue. I established the actual point of indifference by noting which loss amount resulted in equal likelihoods that a coach would decline or accept.
Coaches do really well, but it appears they look for greater losses than required before declining. In other words, they should be declining slightly more often on smaller losses. It's not a big difference, and certainly nothing that will swing any one game. But every little bit helps.
There are additional considerations as well, such as game variables like time and score. For instance, if trailing late in a game a defense would prefer never to hand the offense an additional down with which it could burn 45 more seconds. (DET declined a 10-yd holding penalty on their way to their improbable comeback over DAL last Sunday.) One big consideration is forcing the offense outside of field goal range. But as we saw in the 1st down analysis, that's usually secondary to minimizing the chance of conversion.