Why Calibration Matters
A logistic model assigns each observation a fitted event probability. Those probabilities may be useful even when the goal is not to label each individual as an event or non-event. For example, a hypothetical insurer might use predicted claim probabilities to plan aggregate staffing, while a classification rule would assign each policy a label using a chosen threshold. These are different tasks: calibration asks whether predicted probabilities agree with event frequencies; classification asks whether threshold-based labels match observed outcomes.
A model can classify many observations correctly yet systematically overstate or understate their probabilities. Conversely, probabilities can agree with frequencies across groups even though a particular threshold produces many classification errors. A calibration assessment therefore complements, rather than replaces, the confusion-matrix measures used to assess classifications.
For a logistic regression with predictor vector \(\mathbf{x}_i\), the fitted probability has the form
Here \(Y_i=1\) denotes an event and \(Y_i=0\) denotes no event. The model expresses the event probability conditional on the predictors through the logistic function. Calibration checks compare those probabilities with outcomes; they do not, by themselves, establish that the predictors cause the outcome or that the model is correctly specified in every respect.
Grouped Calibration Tables and Plots
A common descriptive assessment sorts observations by fitted probability and divides them into groups, often called bins. For group \(g\), let \(n_g\) be its size and let \(G_g\) denote the observations in it. Define the observed event count \(O_g\), expected event count \(E_g\), mean fitted probability \(\bar p_g\), and observed event frequency \(\bar y_g\) as follows:
The group’s expected count is the sum of individual fitted probabilities, not necessarily the group size multiplied by one common probability. The latter calculation is equivalent only when all fitted probabilities in the group are equal, or when the common probability used is their exact average.
A grouped calibration table reports at least \(n_g\), \(\bar p_g\), and \(\bar y_g\). It is also useful to show \(E_g\) and \(O_g\), which make the count comparison explicit. In a calibration plot, the horizontal coordinate is typically \(\bar p_g\), and the vertical coordinate is \(\bar y_g\). The identity line, where the two coordinates are equal, represents perfect agreement. Points above the line indicate that the observed frequency in that group exceeded the average fitted probability; points below indicate the reverse.
Worked Example: Building and Reading a Calibration Table
A hypothetical model assigns probabilities to 80 policy records. The records have already been sorted into four groups of 20. Their mean fitted probabilities are \(0.10\), \(0.30\), \(0.60\), and \(0.80\), and the observed event counts are 1, 7, 11, and 18. Construct the group comparisons and describe the calibration pattern.
Method. For each group, calculate \(E_g=n_g\bar p_g\), \(\bar y_g=O_g/n_g\), and the difference \(\bar y_g-\bar p_g\). Since the mean fitted probability is supplied for each group, multiplying it by the group size gives the sum of fitted probabilities.
| Group | \(n_g\) | \(\bar p_g\) | \(E_g\) | \(O_g\) | \(\bar y_g\) | \(\bar y_g-\bar p_g\) |
|---|---|---|---|---|---|---|
| 1 | 20 | 0.10 | 2 | 1 | 0.05 | -0.05 |
| 2 | 20 | 0.30 | 6 | 7 | 0.35 | 0.05 |
| 3 | 20 | 0.60 | 12 | 11 | 0.55 | -0.05 |
| 4 | 20 | 0.80 | 16 | 18 | 0.90 | 0.10 |
| Total | 80 | 36 | 37 |
For example, in group 3 the expected count is \(20(0.60)=12\), while the observed frequency is \(11/20=0.55\). Its plotted point is \((0.60,0.55)\), slightly below the identity line. The corresponding points for groups 1, 2, and 4 are \((0.10,0.05)\), \((0.30,0.35)\), and \((0.80,0.90)\).
Across all groups, the total expected count is \(2+6+12+16=36\), and the observed count is \(1+7+11+18=37\). Thus the overall observed frequency is \(37/80=0.4625\), compared with an overall mean fitted probability of \(36/80=0.4500\). The close overall totals do not imply that every group is calibrated: group 4, for instance, has an observed frequency 0.10 above its mean fitted probability.
Check. The group sizes sum to \(80\), the group expected counts sum to \(36\), and the observed counts sum to \(37\), agreeing with the overall calculations. The common mistake avoided is using only the overall event rate and missing group-level departures.
Why the Expected Count Is the Sum of Probabilities
The expected-count formula follows from the conditional mean of each binary outcome. For a precise conditional argument, let \(\mathcal{X}\) denote the information in the full set of predictors for all observations being considered. Assume that for every observation \(i\), its conditional event probability given this same information is \(p_i\):
For a fixed group \(G_g\), linearity of conditional expectation then gives the result. Independence is not needed for this expectation calculation.
The conditioning information must be the same for every term in the sum. Knowing only that \(E[Y_i\mid X_i]=p_i\) does not, by itself, justify replacing \(E[Y_i\mid\mathcal{X}]\) by \(p_i\) when \(\mathcal{X}\) also includes other observations’ predictors. The stated conditional-mean assumption makes the step valid.
If, in addition, the outcomes are conditionally independent given \(\mathcal{X}\), then the conditional variance of the count is
Each binary outcome has conditional variance \(p_i(1-p_i)\); conditional independence makes the covariances between distinct outcomes zero. Without conditional independence, the variance also includes the pairwise conditional covariances, so the displayed variance formula need not apply. This distinction matters when interpreting unusually large count differences: a standardized discrepancy based on the independence formula is not automatically a formal test of calibration.
Worked Example: Interpreting a Group Count Difference
In a hypothetical validation sample, a group contains 10 observations with fitted probability \(0.20\) and 10 with fitted probability \(0.60\). Twelve events are observed in the group. Assume the outcomes are conditionally independent given the full set of predictors. Calculate the expected count, its conditional standard deviation, and the observed count’s standardized difference from expectation.
Method. Sum the individual probabilities to find the expected count. Under the stated independence assumption, sum the Bernoulli variances and take the square root. Then divide the observed-minus-expected count by that standard deviation.
The expected count is
The conditional variance is
Therefore the conditional standard deviation is \(\sqrt{4.00}=2.00\). The standardized difference is
The group’s average fitted probability is \(8/20=0.40\), while its observed frequency is \(12/20=0.60\). The count is two conditional standard deviations above expectation under the stated variance calculation. This is a descriptive signal to investigate, not by itself a definitive conclusion: grouping, model fitting, dependence, and sampling variability affect how such a discrepancy should be interpreted.
Check. The group has 20 observations and an expected count of 8, so its expected rate is \(0.40\). The observed count of 12 gives rate \(0.60\). The common mistake avoided is treating the two fitted probabilities as if every observation had probability \(0.20\), or calculating the variance as though all 20 probabilities were identical.
Calibration Is Not Classification Accuracy
A classification rule converts a probability into a predicted label by choosing a threshold \(c\): predict an event when \(p_i\ge c\), and a non-event otherwise. Changing \(c\) can change the confusion matrix and classification measures, such as sensitivity, specificity, and accuracy. It does not change the fitted probabilities or the calibration table.
Consequently, classification accuracy cannot establish calibration. Accuracy depends on the threshold and on the observed labels. Calibration concerns whether probabilities, considered as probabilities, correspond to event frequencies. The following example shows that two probability assignments can yield the same classifications and accuracy but very different calibration.
Worked Example: Same Accuracy, Different Calibration
Consider 20 hypothetical records: among 10 records in a low-risk group, 2 events occur; among 10 records in a high-risk group, 8 events occur. Model A assigns probability \(0.20\) to each low-risk record and \(0.80\) to each high-risk record. Model B assigns \(0.40\) and \(0.60\), respectively. Compare grouped calibration and classification at threshold \(c=0.50\).
Grouped calibration. The observed frequencies are \(2/10=0.20\) in the low-risk group and \(8/10=0.80\) in the high-risk group. Model A predicts exactly those group frequencies: its expected counts are \(10(0.20)=2\) and \(10(0.80)=8\).
Model B’s expected counts are \(10(0.40)=4\) and \(10(0.60)=6\). It overpredicts the low-risk group by \(4-2=2\) events and underpredicts the high-risk group by \(8-6=2\) events. Its total expected count is \(4+6=10\), equal to the total observed count \(2+8=10\). Thus Model B is calibrated in the aggregate across all 20 records, but not in these two groups.
Classification at \(0.50\). Both models label every low-risk record as a non-event and every high-risk record as an event. In the high-risk group, 8 events are true positives and 2 non-events are false positives. In the low-risk group, 8 non-events are true negatives and 2 events are false negatives. Both models therefore have accuracy
The two models have identical classifications and accuracy at this threshold, yet Model A matches both group frequencies while Model B does not. Model B’s equal-and-opposite group count differences also show why an aggregate comparison alone can conceal miscalibration.
Check. The actual event count is \(2+8=10\), and both models’ total expected count is 10. The same confusion-matrix counts follow because the predictions fall on the same side of \(0.50\). The common mistake avoided is concluding that equal accuracy, or even equal total observed and expected counts, means equal calibration.
Practical Interpretation and Common Traps
Grouped calibration is a useful summary, but it depends on how observations are grouped. Small groups can have highly variable observed frequencies; large groups can hide meaningful differences among observations with different fitted probabilities. Different bin boundaries can also produce different summaries. State how groups were formed and report group sizes alongside frequencies so that a rate based on a few observations is not given the same visual weight as a rate based on many.
A calibration plot is descriptive. The identity line is a reference, not a guarantee that finite-sample points should lie exactly on it. Points may deviate because of ordinary outcome variation. If probabilities are assessed on the same data used to estimate the model, the assessment can also reflect the fitting process; a separate validation sample is often useful when the objective is to evaluate performance on new observations.
- Do not compare only observed count with one rounded probability. Calculate \(E_g=\sum p_i\), or use the exact group mean probability multiplied by \(n_g\).
- Do not confuse threshold labels with probability calibration. A threshold changes classifications, not fitted probabilities.
- Do not treat aggregate agreement as proof of group agreement. Overprediction in one group can cancel underprediction in another.
- Do not interpret every plotted departure as a systematic model defect. Group frequencies have sampling variation, and binning choices affect the display.
- Do not use a variance formula without its assumptions. The sum of Bernoulli variances requires conditional independence for the grouped count.
Check Your Understanding
For each problem, distinguish the quantities being compared and retain the stated grouping or threshold.
- A group has 40 observations, mean fitted probability \(0.25\), and 12 observed events. Find its expected count, observed frequency, and difference between observed and fitted frequencies. (Answer: expected count \(10\); observed frequency \(0.3000\); difference \(0.0500\).)
- Two groups have sizes 15 and 25, mean fitted probabilities \(0.10\) and \(0.40\), and observed event counts 1 and 12. Find the total expected count, total observed count, and overall observed and expected rates. (Answer: expected count \(11.5\); observed count \(13\); observed rate \(0.3250\); expected rate \(0.2875\).)
- At threshold \(0.40\), a low-probability group of 20 has mean fitted probability \(0.30\) and 5 events; a high-probability group of 20 has mean fitted probability \(0.70\) and 15 events. The first group’s predictions are below the threshold and the second group’s predictions are at or above it. Find the two observed frequencies and classification accuracy. (Answer: observed frequencies \(0.2500\) and \(0.7500\); accuracy \(0.7500\).)
- A group consists of six observations with fitted probability \(0.10\) and four with fitted probability \(0.50\). Five events are observed. Assuming conditional independence given the full predictor set, find the expected count, conditional variance, and standardized difference. (Answer: expected count \(2.6\); variance \(1.54\); standardized difference \(2.4/\sqrt{1.54}\approx1.934\).)
- In each of two groups of 10, the mean fitted probability is \(0.20\) and \(0.60\), respectively. The observed event counts are 4 and 4. Compare aggregate and group-level calibration. (Answer: total expected and observed counts are both \(8\), so aggregate rates both equal \(0.4000\); group frequencies are \(0.4000\) versus \(0.2000\), and \(0.4000\) versus \(0.6000\), so the groups show opposite calibration discrepancies.)