Why Bin Residuals for a Binary Response?
In logistic regression, each response \(Y_i\) is either 0 or 1, and the fitted probability \(\hat p_i\) generally differs across observations. A plot of the individual residuals \(Y_i-\hat p_i\) can be difficult to read: each response has only two possible values, so the residuals form two bands rather than a smooth cloud. Grouping observations with similar fitted probabilities helps reveal whether event rates are systematically higher or lower than the model expects.
A binned residual plot is a diagnostic, not a replacement for checking the model assumptions or a formal goodness-of-fit test. It summarizes calibration across ranges of fitted risk: among observations assigned similar probabilities, does the observed proportion of events broadly agree with the model’s predictions?
The expected rate is the average of the individual fitted probabilities in that bin. The corresponding expected number of events is \(\sum_{i\in B}\hat p_i\). These quantities are related by dividing the expected number by \(n_B\). A plot may show observed rate against expected rate, or show \(r_B\) against a bin summary such as \(\bar p_B\). In the first version, perfect calibration lies along the line of equality; in the second, perfect calibration lies along the horizontal line \(r_B=0\).
Constructing and Reading the Plot
A common construction sorts the observations by \(\hat p_i\) and partitions them into bins with approximately equal numbers of observations. For example, 100 observations might be divided into 10 consecutive bins of 10. Equal-sized bins make the amount of information in each point easier to compare. Bins can instead be chosen to reflect meaningful risk ranges, but the choice should be stated because it can change the appearance of the plot.
For each bin, calculate both the observed event rate and the average fitted probability. Plot the pair \((\bar p_B,\bar Y_B)\), or plot \((\bar p_B,r_B)\). Points consistently above the equality line in the first plot, or above zero in the residual plot, indicate that observed event rates exceed model predictions in those ranges. Points below indicate that predictions tend to exceed observed rates. A curved or otherwise systematic pattern can suggest that the model’s predictions are not calibrated uniformly across fitted risk.
The bin average matters when fitted probabilities differ within a bin. It is not generally correct to represent the bin by the fitted probability of a “typical” individual, or by applying the logistic function to an average predictor value. The expected number of events is obtained by adding the individual probabilities; the expected rate is that sum divided by the bin size.
Worked Example: Calculating Four Equal-Probability Bin Summaries
Method. For each bin, compare the observed event rate with the common fitted probability. A hypothetical insurer’s model assigns the same fitted probability to all 20 policies in each of four bins. The numbers of observed claims in the bins are 1, 7, 8, and 16, respectively. Compute the plotted values, their approximate standard errors, and standardized residuals. This calculation avoids confusing a count difference with a rate difference.
For the first bin, \(n_B=20\), \(\hat p_i=0.10\), and there is 1 event. Thus $$ \bar Y_B=\frac{1}{20}=0.05,\qquad \bar p_B=0.10,\qquad r_B=0.05-0.10=-0.05. $$
For the second bin, the event rate is \(7/20=0.35\), the expected rate is 0.30, and the residual is \(0.35-0.30=0.05\). For the third bin, these values are \(8/20=0.40\), 0.50, and \(0.40-0.50=-0.10\). For the fourth bin, they are \(16/20=0.80\), 0.70, and \(0.80-0.70=0.10\).
Under the Bernoulli model, the approximate standard error of the rate difference in a bin is calculated below. The standardized residual is \(r_B/\operatorname{SE}(r_B)\).
| Fitted probability | Events / 20 | Observed rate | Expected rate | Residual | Standard error | Standardized residual |
|---|---|---|---|---|---|---|
| 0.10 | 1 / 20 | 0.05 | 0.10 | -0.05 | 0.0671 | -0.745 |
| 0.30 | 7 / 20 | 0.35 | 0.30 | 0.05 | 0.1025 | 0.488 |
| 0.50 | 8 / 20 | 0.40 | 0.50 | -0.10 | 0.1118 | -0.894 |
| 0.70 | 16 / 20 | 0.80 | 0.70 | 0.10 | 0.1025 | 0.976 |
For example, in the first bin the standard error is \(\sqrt{0.10(1-0.10)/20}=0.0671\), and the standardized residual is \(-0.05/0.0671=-0.745\). The other standard errors and standardized residuals follow the same calculation. All four standardized residuals are between \(-2\) and 2, so none is individually large by that rough reference. This does not prove that the model is correct; the plot should also be assessed for a systematic pattern.
Check. The observed number of events across the four bins is \(1+7+8+16=32\). The model’s expected number is \(20(0.10+0.30+0.50+0.70)=32\). The total counts agree, even though individual bin residuals are nonzero. Aggregating all bins could therefore hide local differences.
Why the Standard Error Has This Form
For uncertainty calculations, first consider the Bernoulli model with probabilities \(p_i\) treated as fixed. Conditional on the predictors, assume the responses are independent and \(Y_i\sim\operatorname{Bernoulli}(p_i)\). Then \(E[Y_i]=p_i\) and \(\operatorname{Var}(Y_i)=p_i(1-p_i)\). The bin residual is the average of the individual residuals \(Y_i-p_i\).
This formula also explains why equal-sized bins need not have equal standard errors: the Bernoulli variance \(p_i(1-p_i)\) depends on the fitted probabilities. If every observation in a bin has the same fitted probability \(\hat p\), the formula simplifies to \(\sqrt{\hat p(1-\hat p)/n_B}\).
A rough visual reference is to compare residuals with about two standard errors in either direction. It is only a guide. The probabilities in an actual diagnostic are estimated from the same data, not known constants; parameter estimation can affect the variability of the residuals. In addition, several bins are inspected and the bin boundaries may be chosen after looking at fitted probabilities. Consequently, these bands are not a formal test with a guaranteed overall error rate.
Worked Example: A Bin With Unequal Fitted Probabilities
Method. Add the fitted probabilities to obtain the expected event count, and add the Bernoulli variances to obtain the uncertainty. A hypothetical group of five applicants has fitted probabilities \(0.10, 0.20, 0.30, 0.60,\) and \(0.80\). Their observed outcomes are \(0,1,0,1,\) and \(1\). Calculate the bin summaries and standardized residual. This avoids substituting a single probability for heterogeneous predictions.
There are three observed events, so the observed rate is $$ \bar Y_B=\frac{0+1+0+1+1}{5}=\frac{3}{5}=0.60. $$ The sum of fitted probabilities is \(0.10+0.20+0.30+0.60+0.80=2.00\). Thus the expected number of events is 2.00 and the expected rate is \(2.00/5=0.40\). The bin residual is \(0.60-0.40=0.20\).
The variance sum is $$ 0.10(0.90)+0.20(0.80)+0.30(0.70)+0.60(0.40)+0.80(0.20) =0.09+0.16+0.21+0.24+0.16=0.86. $$ Therefore, the standard error of the rate difference is \(\sqrt{0.86}/5=0.1855\), and the standardized residual is \(0.20/0.1855=1.078\), to three decimals.
Check. On the event-count scale, the observed-minus-expected count is \(3-2=1\), and its standard deviation is \(\sqrt{0.86}=0.9274\). The standardized difference is \(1/0.9274=1.078\), consistent with the rate-scale calculation. A bin of only five observations is highly variable, so the positive residual alone is weak evidence of a model problem.
Interpreting Patterns Rather Than Isolated Points
Binning trades detail for readability. A large number of bins preserves more local variation but can produce noisy rates, especially where events are rare. A small number of bins gives more stable averages but can conceal a localized mismatch. Unequal bin sizes also make uncertainty differ across points, so a visually large residual in a small bin may be less informative than it first appears.
Look for broad structure: a sequence of residuals mostly below zero at low fitted probabilities and above zero at high fitted probabilities, for example, suggests that the model tends to overpredict at the low end and underpredict at the high end. A single unusual point may instead reflect sampling variation. A plot should be read alongside the data context and other diagnostics, rather than treated as a pass-or-fail rule.
Worked Example: Reading a Gradual Residual Pattern
Method. Compare the observed and expected rates across ordered bins, then check the size of each difference relative to its standard error. A hypothetical model has five bins of 25 accounts each. The fitted probability is constant within each bin, with values \(0.10, 0.25, 0.40, 0.55,\) and \(0.70\). The observed event counts are \(0,4,9,16,\) and \(21\). Identify the pattern and assess whether an individual bin has a standardized residual exceeding 2 in absolute value. This avoids interpreting a visual trend as a formal rejection by itself.
The observed rates are \(0/25=0.00\), \(4/25=0.16\), \(9/25=0.36\), \(16/25=0.64\), and \(21/25=0.84\). Subtracting the fitted rates gives residuals \(-0.10,-0.09,-0.04,0.09,\) and \(0.14\). The first three are negative and the last two are positive, so the direction of the discrepancies shifts from overprediction in the lower fitted-probability bins toward underprediction in the higher bins.
The standard errors are calculated as \(\sqrt{\hat p(1-\hat p)/25}\) because fitted probabilities are constant within each bin. They are \(0.0600,0.0866,0.0980,0.0995,\) and \(0.0917\). Dividing residuals by these values gives standardized residuals \(-1.667,-1.039,-0.408,0.905,\) and \(1.527\), respectively. None exceeds 2 in absolute value.
Interpretation and check. The ordered signs suggest a possible calibration pattern worth investigating, but no bin has a large standardized residual by the rough two-standard-error reference. The plot is not conclusive: binning can obscure details, and estimated probabilities make the reference approximate.
Common Mistake: Comparing Counts With Rates
Suppose a bin contains 20 observations, has 7 observed events, and has average fitted probability 0.30. A common error is to calculate \(7-0.30=6.70\) and call this the residual. This subtracts a rate from a count, so the result has no consistent interpretation.
On the rate scale, the observed rate is \(7/20=0.35\), and the binned residual is \(0.35-0.30=0.05\). On the count scale, the expected count is \(20(0.30)=6\), so the observed-minus-expected count is \(7-6=1\). These results agree after dividing the count difference by 20: \(1/20=0.05\). Always compare quantities on the same scale.
Check Your Understanding
For each problem, calculate or interpret the requested bin summary before checking the answer.
- A bin contains 10 observations, with 3 events and fitted probability 0.20 for every observation. Find the observed rate, expected rate, and binned residual. (Answer: observed rate \(0.30\), expected rate \(0.20\), residual \(0.10\).)
- A bin has four fitted probabilities \(0.10,0.30,0.50,0.70\), and observed outcomes \(0,0,1,1\). Find the expected rate, observed rate, residual, and approximate standard error. (Answer: expected rate \(0.40\), observed rate \(0.50\), residual \(0.10\), standard error \(\sqrt{0.76}/4\approx0.2184\).)
- A bin has 40 observations, each with fitted probability \(0.25\), and 14 observed events. Find its rate residual and standardized residual using the Bernoulli standard error. (Answer: residual \(14/40-0.25=0.10\); standard error \(\sqrt{0.25(0.75)/40}\approx0.0685\); standardized residual \(\approx1.461\).)
- Across bins ordered by fitted probability, residuals are mostly negative in the lower bins and mostly positive in the higher bins. State the direction of the discrepancy and one reason not to treat the pattern alone as a formal test. (Answer: the model tends to overpredict event rates at the low end and underpredict at the high end; the pattern is subject to sampling variation, bin choice, and parameter estimation.)
- A bin of 12 observations has 5 events and an average fitted probability of \(0.40\). A candidate reports \(5-0.40=4.60\) as the residual. Correct the calculation on both the rate and count scales. (Answer: rate residual \(5/12-0.40\approx0.0167\); expected count \(12(0.40)=4.8\), so count residual \(5-4.8=0.2\), which divided by 12 is approximately \(0.0167\).)