What a Binned Calibration Table Summarizes
A binary GLM assigns each observation a predicted event probability. A calibration table groups observations with similar predicted probabilities and, within each group, compares the average prediction with the observed event proportion. The comparison is useful as a compact description, but the choice of groups affects what the table shows.
Let \(Y_i\in\{0,1\}\) be the observed outcome for observation \(i\), and let \(\hat p_i\in[0,1]\) be its predicted probability. A value of \(Y_i=1\) denotes an event. For a specified partition of the predictions into \(K\) nonempty bins, let \(B_k\) be the indices in bin \(k\), and let \(m_k\) be its number of observations.
The table is a descriptive grouping of predictions and outcomes. Its observed proportions are not individual probabilities, and a small gap does not establish that every prediction in the bin is accurate. The averaging can conceal differences among observations within the same bin.
Choosing Bins
In equal-width binning, cutpoints divide the probability scale into intervals of the same width. For example, five bins of width \(0.2\) might be \([0,0.2)\), \([0.2,0.4)\), \([0.4,0.6)\), \([0.6,0.8)\), and \([0.8,1]\). The last interval includes 1 so every valid predicted probability is assigned to a bin.
In equal-count binning, observations are ordered by predicted probability and divided into groups with approximately the same number of observations. The intervals in predicted-probability units will generally have different widths. Tied predictions require care: splitting observations with the same prediction across bins can make the table harder to interpret. A common alternative is to keep ties together and accept unequal bin sizes.
Neither rule is universally best. Equal-width bins give intervals that are easy to interpret on the probability scale, but may leave some bins sparsely populated or empty. Equal-count bins avoid some sparse-bin problems, but may combine a narrow range of predictions in one bin and a much wider range in another. Both depend on the sample and on choices made when setting cutpoints.
Worked Example: Constructing an Equal-Width Table
Suppose a fitted binary GLM gives the following predictions and the outcomes are as shown. The observations are listed in increasing order of predicted risk.
| Observation | Predicted risk \(\hat p_i\) | Outcome \(Y_i\) |
|---|---|---|
| 1 | 0.05 | 0 |
| 2 | 0.12 | 0 |
| 3 | 0.18 | 1 |
| 4 | 0.28 | 0 |
| 5 | 0.42 | 0 |
| 6 | 0.49 | 1 |
| 7 | 0.51 | 0 |
| 8 | 0.62 | 1 |
| 9 | 0.78 | 1 |
| 10 | 0.92 | 1 |
Use five equal-width bins, with the intervals defined above. For each row, count the observations and events, sum the predictions, and divide the event count and prediction sum by the bin size. The gap is calculated from the unrounded quantities.
| Prediction bin | Count | Events | Observed \(O_k\) | Mean predicted \(\bar p_k\) | Gap \(D_k\) |
|---|---|---|---|---|---|
| [0, 0.2) | 3 | 1 | 1/3 = 0.3333 | 0.35/3 = 0.1167 | 0.2167 |
| [0.2, 0.4) | 1 | 0 | 0 | 0.28 | -0.2800 |
| [0.4, 0.6) | 3 | 1 | 1/3 = 0.3333 | 1.42/3 = 0.4733 | -0.1400 |
| [0.6, 0.8) | 2 | 2 | 1 | 1.40/2 = 0.7000 | 0.3000 |
| [0.8, 1] | 1 | 1 | 1 | 0.92 | 0.0800 |
For example, the first bin contains outcomes \(0,0,1\), so its event count is 1 and its observed proportion is \(1/3\). Its predictions sum to \(0.05+0.12+0.18=0.35\), giving a mean of \(0.35/3=0.1167\) and a gap of \(1/3-0.35/3=0.2167\). In the middle bin, the outcomes are \(0,1,0\), so the event count is 1, not 2. Its gap is \(1/3-1.42/3=-0.1400\).
The second row has only one observation, and therefore its observed proportion is either 0 or 1. Its gap of \(-0.28\) is a correct calculation for this sample, but that single observation provides little basis for judging calibration across the probability interval.
Comparing Equal-Width and Equal-Count Bins
The same predictions can yield tables with different bin memberships under different rules. That changes both the bin-specific averages and the amount of data supporting each observed proportion. The next calculation uses the same ten observations to make the comparison direct.
Worked Example: Rebinning into Equal-Count Groups
Divide the ordered observations into two groups of five. The first group contains observations 1–5. It has one event, and its predictions sum to \(0.05+0.12+0.18+0.28+0.42=1.05\). Thus \(O_1=1/5=0.20\), \(\bar p_1=1.05/5=0.21\), and \(D_1=0.20-0.21=-0.01\).
The second group contains observations 6–10. Its outcomes are \(1,0,1,1,1\), giving four events. Its predictions sum to \(0.49+0.51+0.62+0.78+0.92=3.32\). Thus \(O_2=4/5=0.80\), \(\bar p_2=3.32/5=0.664\), and \(D_2=0.80-0.664=0.136\).
| Equal-count group | Prediction range | Count | Observed \(O_k\) | Mean predicted \(\bar p_k\) | Gap \(D_k\) |
|---|---|---|---|---|---|
| 1 | 0.05 to 0.42 | 5 | 0.200 | 0.210 | -0.010 |
| 2 | 0.49 to 0.92 | 5 | 0.800 | 0.664 | 0.136 |
This table has no one-observation bins, but each group spans a broad prediction range. Its summaries are not a correction to the equal-width table: they answer a question about different sets of observations. In the equal-width table, the bin \([0.2,0.4)\) contained only the prediction \(0.28\); in the equal-count table, that observation is grouped with predictions from \(0.05\) to \(0.42\).
Why Small Bins Are Noisy
An observed proportion based on few outcomes can change substantially when even one outcome changes. This is a sampling property, not a peculiarity of GLMs. The following result gives a simple quantitative bound under explicit assumptions.
Proof. Since \(O_k=m_k^{-1}\sum_{i\in B_k}Y_i\), linearity of expectation gives \(E[O_k]=m_k^{-1}\sum_{i\in B_k}E[Y_i]=m_k^{-1}\sum_{i\in B_k}q_i\). Independence makes the covariance terms in the variance of the sum equal to zero. Also, a Bernoulli variable with probability \(q_i\) has variance \(q_i(1-q_i)\). Therefore
For every \(q_i\in[0,1]\), \(q_i(1-q_i)=1/4-(q_i-1/2)^2\leq1/4\). Summing this bound over the \(m_k\) observations gives \(\operatorname{Var}(O_k)\leq m_k/(4m_k^2)=1/(4m_k)\). Thus the bound decreases as bin size increases. \(\square\)
This result concerns the variability of the observed proportion, not the variability of the gap in every fitted-data setting. It treats the predictions as fixed and the outcomes as independent. When a model is fitted using the same outcomes being summarized, the predictions are data-dependent; the simple fixed-prediction calculation should not be treated as a complete uncertainty analysis for the fitted table.
Cutpoints Can Change the Story
A binning rule can make an apparent pattern more or less prominent. The following example keeps the data unchanged and changes only a cutpoint. It illustrates why a table should report its bin definitions and why a visual impression from a few bins should not be treated as a definitive conclusion.
Worked Example: Sensitivity to a Cutpoint
First use two bins separated at \(0.50\): predictions below \(0.50\), and predictions at least \(0.50\). The lower bin contains observations 1–6. It has two events, so \(O=2/6=0.3333\). The prediction sum is \(1.54\), so \(\bar p=1.54/6=0.2567\), and the gap is \(0.0767\).
The upper bin contains observations 7–10. It has three events, so \(O=3/4=0.7500\). Its prediction sum is \(0.51+0.62+0.78+0.92=2.83\), giving \(\bar p=2.83/4=0.7075\) and a gap of \(0.0425\).
Now move the cutpoint to \(0.60\). The lower bin contains observations 1–7, with two events. Its prediction sum is \(2.05\), so \(O=2/7=0.2857\), \(\bar p=2.05/7=0.2929\), and \(D=-0.0071\). The upper bin contains observations 8–10, all events. Its prediction sum is \(0.62+0.78+0.92=2.32\), so \(O=3/3=1\), \(\bar p=2.32/3=0.7733\), and \(D=0.2267\).
The first cut gives two positive gaps, while the second gives a near-zero gap in the lower bin and a much larger positive gap in the upper bin. The outcomes and predictions did not change; the grouping did. The upper bin after the second cut has only three observations, so its observed proportion of 1 is also a small-sample summary, not evidence that the event probability for every observation in that range is 1.
What the Table Does—and Does Not—Establish
A calibration table is useful for communicating how observed event frequencies compare with average predictions over stated ranges. It can reveal a pattern worth examining and can make a large set of predictions easier to summarize. But the table is not independent of the analyst’s binning decisions, and its summaries do not describe variation within bins.
There is a useful algebraic interpretation of each gap:
Derivation. Substitute the definitions and put the two averages over the same denominator:
So a gap is the average observed-minus-predicted residual in that bin. Positive and negative residuals can cancel, producing a small gap even when individual residuals are not small. Also, the overall average residual can be small while particular bins have gaps of opposite signs. The identity is exact, but it does not turn a grouped summary into a statement about each observation.
A common mistake is to choose whichever binning rule gives the smallest or most reassuring gaps and then treat those gaps as proof of good calibration. That reasoning overlooks that changing the bins changes the averages being compared. A better practice is to specify the rule and cutpoints, show bin counts, observed proportions, and mean predictions together, and interpret sparse bins cautiously. In particular, a one-observation bin’s observed proportion is necessarily 0 or 1, regardless of whether its prediction is close to that value.
Check Your Understanding
Use the definitions and examples to answer the following questions.
- A bin contains four observations, with outcomes \(0,1,0,1\) and predictions \(0.10,0.30,0.40,0.60\). Calculate its observed proportion, mean predicted risk, and calibration gap.
- What is one advantage and one limitation of equal-count binning compared with equal-width binning?
- Why can a one-observation bin give a visually large gap even when that observation alone does not establish poor calibration?
- State the assumptions under which the variance bound for \(O_k\) was derived.
- Explain why changing a cutpoint can change a table’s gaps even though none of the observations or predictions change.