Why Binary Residuals Need a Different Reading
For a continuous response, a residual plot may show a cloud of positive and negative values. A binary response has only two possible observed outcomes, so its raw residuals cannot form the same kind of cloud. In a logistic regression, each observation has a fitted event probability, and its raw residual is determined by that probability together with whether the event occurred.
This structure is useful to recognize. The two bands in a residual-versus-fitted-probability plot are a mathematical consequence of binary data, not by themselves evidence that a model is wrong. To look for systematic lack of fit, compare observed event rates with fitted probabilities across meaningful groups and inspect whether the differences show a coherent pattern.
For this interpretation, assume the response is recorded as 0 or 1 and the fitted probability is between 0 and 1. In a correctly specified model for the conditional event probability, \(E[Y_i\mid \mathbf{x}_i]=p_i\), where \(\mathbf{x}_i\) denotes the predictors for observation \(i\). The fitted probability \(\hat{p}_i\) is an estimate of that conditional probability. Raw residuals are useful for identifying patterns, but they are not normally distributed with constant variance.
Where the Two Residual Bands Come From
Plot \(r_i\) vertically against \(\hat{p}_i\) horizontally. For every non-event, the residual is \(-\hat{p}_i\). For every event, it is \(1-\hat{p}_i\). Consequently, points lie on two curves: one descends from 0 to -1, and the other descends from 1 to 0. At a fixed fitted probability, the possible raw residuals are only those two values.
The gap between the two possible residuals at a given fitted probability is exactly 1. As \(\hat{p}_i\) increases, a non-event has a more negative residual, while an event has a smaller positive residual. This is expected: a non-event is more surprising when the model predicted a high probability, and an event is less surprising when the model predicted a high probability.
Worked Example: Reading the Two Bands
A fitted binary-response model assigns six hypothetical policy records the following probabilities. The observed event indicators, in the same order, are 0, 1, 0, 1, 1, 0. Calculate each raw residual and explain what the plot would show.
Method. Use \(r_i=Y_i-\hat{p}_i\), applying the non-event formula when \(Y_i=0\) and the event formula when \(Y_i=1\).
| Observation | \(\hat{p}_i\) | \(Y_i\) | Calculation | \(r_i\) |
|---|---|---|---|---|
| 1 | 0.20 | 0 | \(0-0.20\) | -0.20 |
| 2 | 0.30 | 1 | \(1-0.30\) | 0.70 |
| 3 | 0.40 | 0 | \(0-0.40\) | -0.40 |
| 4 | 0.60 | 1 | \(1-0.60\) | 0.40 |
| 5 | 0.70 | 1 | \(1-0.70\) | 0.30 |
| 6 | 0.80 | 0 | \(0-0.80\) | -0.80 |
The three non-events have residuals \(-0.20\), \(-0.40\), and \(-0.80\), each equal to minus its fitted probability. The three events have residuals \(0.70\), \(0.40\), and \(0.30\), each equal to one minus its fitted probability. A residual-versus-fitted plot places these points on the two expected bands. The negative residual of \(-0.80\) is notable because the model assigned that non-event probability 0.80, but one such observation alone does not establish systematic lack of fit.
Check. For each row, adding the residual to the fitted probability recovers the observed indicator: \(r_i+\hat{p}_i=Y_i\). The calculation also avoids the common mistake of giving a non-event residual the positive value \(\hat{p}_i\).
Grouped Residual Summaries
Because individual binary residuals are constrained to the two bands, a group summary can be easier to interpret. For a group \(G\) of \(n_G\) observations, compare the observed event rate with the average fitted probability. These quantities have a direct residual interpretation.
The equality follows by substituting \(r_i=Y_i-\hat{p}_i\) into the average and distributing the sum. Thus a positive group mean residual means the observed event rate exceeds the average fitted probability in that group; a negative value means it falls below. This is a compact way to summarize whether the model tends to underpredict or overpredict events in a part of the data.
Worked Example: Comparing Event Rates Across Risk Groups
A hypothetical model is applied to three groups of five applicants, grouped by a risk characteristic. For each group, the fitted probabilities are listed in the table. The observed event counts are 1, 3, and 4, respectively. Calculate the average fitted probability, observed event rate, and mean raw residual for each group.
Method. Sum the fitted probabilities to get the expected number of events, divide by five to obtain the average fitted probability, and subtract that average from the observed event rate.
| Group | Fitted probabilities | Expected events | Observed rate | Mean residual |
|---|---|---|---|---|
| A | 0.10, 0.20, 0.20, 0.30, 0.20 | 1.00 | 1/5 = 0.20 | 0.20 - 0.20 = 0.00 |
| B | 0.30, 0.40, 0.40, 0.50, 0.40 | 2.00 | 3/5 = 0.60 | 0.60 - 0.40 = 0.20 |
| C | 0.60, 0.60, 0.70, 0.70, 0.90 | 3.50 | 4/5 = 0.80 | 0.80 - 0.70 = 0.10 |
For group A, the fitted probabilities sum to \(0.10+0.20+0.20+0.30+0.20=1.00\), so their average is \(1.00/5=0.20\). The observed rate is \(1/5=0.20\), giving a mean residual of 0.00.
For group B, the fitted probabilities sum to \(0.30+0.40+0.40+0.50+0.40=2.00\). Their average is \(2.00/5=0.40\). The observed rate is \(3/5=0.60\), giving a mean residual of \(0.60-0.40=0.20\).
For group C, the fitted probabilities sum to \(0.60+0.60+0.70+0.70+0.90=3.50\). Their average is \(3.50/5=0.70\). The observed rate is \(4/5=0.80\), giving a mean residual of \(0.80-0.70=0.10\).
The model matches group A's event rate and underpredicts the event rates in groups B and C. That pattern is a diagnostic signal to investigate, not a formal conclusion that the model is inadequate: each group contains only five observations, so observed rates can vary substantially. As a check, the group B sum of raw residuals is \(3-2.00=1.00\), and dividing by five gives the mean residual \(0.20\).
Fitted-Probability Displays
A fitted-probability display compares grouped observed event rates with grouped average fitted probabilities. One axis shows \(\bar{p}_G\); the other shows \(\bar{Y}_G\). The 45-degree line represents agreement: groups on the line have observed rates equal to their average predictions. Points above the line indicate underprediction, and points below it indicate overprediction.
Groups can be formed from ranges of a predictor when the diagnostic question concerns that predictor. Alternatively, sort observations by fitted probability and make groups of roughly similar size. The second approach examines calibration across the range of model-predicted risk. In either case, display the group sizes as well as the points: an observed rate from a small group is less stable than one based on many observations.
Worked Example: Interpreting a Calibration Pattern
A hypothetical insurer groups 100 records into five equal-sized groups by fitted probability. The group summaries are shown below. Calculate the differences between observed and fitted rates, then describe the pattern.
| Group | Average fitted probability | Observed events | Observed rate | Observed minus fitted |
|---|---|---|---|---|
| 1 | 0.10 | 4 of 20 | 0.20 | 0.20 - 0.10 = 0.10 |
| 2 | 0.30 | 7 of 20 | 0.35 | 0.35 - 0.30 = 0.05 |
| 3 | 0.50 | 9 of 20 | 0.45 | 0.45 - 0.50 = -0.05 |
| 4 | 0.70 | 12 of 20 | 0.60 | 0.60 - 0.70 = -0.10 |
| 5 | 0.90 | 15 of 20 | 0.75 | 0.75 - 0.90 = -0.15 |
The observed rates are computed as \(4/20=0.20\), \(7/20=0.35\), \(9/20=0.45\), \(12/20=0.60\), and \(15/20=0.75\). Subtracting the corresponding fitted probabilities gives \(0.10\), \(0.05\), \(-0.05\), \(-0.10\), and \(-0.15\).
The first two groups lie above the agreement line, while the last three lie below it. In these groups the model underpredicts at the lower fitted probabilities and overpredicts at the higher ones. This orderly pattern is more informative than a single large raw residual and suggests that calibration across the risk range deserves further investigation. It does not, on its own, identify the cause or prove that the model is wrong.
Check. Each observed rate is between 0 and 1, and each difference equals observed rate minus average fitted probability. The group counts sum to 100, matching the stated number of records.
How to Use the Displays Carefully
Grouped summaries reduce the visual distraction of individual binary outcomes, but grouping is a diagnostic choice rather than a cure for model limitations. Too few groups can hide a local pattern; too many create noisy rates. A pattern that appears only under one arbitrary grouping scheme should be interpreted cautiously. Examine the raw residual plot as well as grouped results, and consider plots against predictors when a particular relationship is in question.
A calibration display summarizes agreement between predicted and observed event rates. It cannot establish that the model has the correct relationship with every predictor, nor can it reveal every form of misspecification. Two observations with similar fitted probabilities may have quite different predictor values. A model can therefore look well calibrated overall while still missing structure associated with a predictor or subgroup.
Confirm that the outcome is coded 0 or 1 and calculate each raw residual as observed indicator minus fitted probability.
Verify that non-events follow \(r_i=-\hat{p}_i\) and events follow \(r_i=1-\hat{p}_i\). Do not treat the two-band shape alone as a warning sign.
For each group, compare the observed event rate with the average fitted probability; their difference is the mean raw residual.
Use group sizes and the direction of departures to judge whether a pattern merits closer examination, while allowing for ordinary variation.
Common Exam Trap
A frequent error is to read a large individual residual as conclusive evidence of model failure. For example, if a model predicts an event probability of 0.80 and the event does not occur, the raw residual is \(0-0.80=-0.80\). That residual is large in magnitude because the observed outcome was surprising under the prediction. But a binary event with probability 0.80 can still fail to occur; the observation alone does not establish systematic lack of fit.
The reverse error is to call the two bands a defect simply because the residuals do not form an unstructured cloud around zero. The bands follow directly from the 0/1 response. The useful diagnostic question is whether residuals or grouped rates show a consistent direction or structure beyond that expected form. A coherent pattern across groups, predictors, or fitted-probability ranges is more compelling than the shape shared by all binary-response residual plots.
Check Your Understanding
For each problem, calculate the requested quantity before checking the answer.
- A record has observed outcome 0 and fitted event probability 0.25. Calculate its raw residual and identify its band. (Answer: \(-0.25\), the non-event band.)
- In a group of three records, the fitted probabilities are 0.20, 0.30, and 0.50, and two events occur. Find the observed rate, average fitted probability, and mean raw residual. (Answer: observed rate \(2/3\), average fitted probability \(1/3\), mean residual \(1/3\).)
- Four records have fitted probabilities 0.10, 0.20, 0.30, and 0.40, with outcomes 1, 0, 1, and 0. Calculate the sum of raw residuals and compare the observed rate with the average fitted probability. (Answer: residual sum \(1.00\); observed rate \(0.50\), average fitted probability \(0.25\), difference \(0.25\).)
- Two equal-sized fitted-probability groups each contain four records. In the lower group every fitted probability is 0.20 and one event occurs; in the higher group every fitted probability is 0.80 and three events occur. Calculate each group's observed-minus-fitted difference and interpret the direction. (Answer: lower group \(0.25-0.20=0.05\), higher group \(0.75-0.80=-0.05\); the model underpredicts in the lower group and overpredicts in the higher group.)
- A group has six records with fitted probabilities summing to 2.40, and three events are observed. Calculate the average fitted probability, observed rate, and mean residual. State whether the group alone proves model lack of fit. (Answer: average fitted probability \(0.40\), observed rate \(0.50\), mean residual \(0.10\); no, a single group summary is a diagnostic signal, not proof.)