Why Binning Choices Matter
A calibration plot summarizes how predicted values compare with observed outcomes. Rather than plotting every case separately, it groups cases with similar predictions and displays a summary for each group. That makes the pattern easier to see, but the picture depends on how the groups are formed. More bins can reveal local changes while leaving fewer cases in each group; fewer bins produce broader summaries that may conceal local differences.
The goal is not to find a single universally correct number of bins. It is to make a display whose summaries are understandable and whose limitations are visible. The grouping should be described, and important conclusions should not depend entirely on one arbitrary choice of boundaries.
The calculations require matched prediction-outcome pairs on the same scale: each case must have one prediction and its corresponding observed outcome. For a descriptive calibration plot, the bins should be specified using the predictions, should not overlap, and should cover the prediction range being summarized. State which endpoints belong to each bin. These summaries describe the evaluated cases; by themselves, they do not establish why predictions and outcomes differ.
For a binary outcome, predictions are often probabilities between 0 and 1. A point at \((0.30,0.40)\), for example, means that the cases in that bin had an average predicted probability of 0.30 and an observed event proportion of 0.40. The diagonal reference line represents equality of those two summaries. The rest of this tutorial focuses on how the binning determines the points, not on fitting a line to them.
Equal-Width and Equal-Count Bins
Two common strategies are equal-width bins and equal-count bins. Equal-width bins cover intervals of the same length on the prediction scale. For example, the ranges from 0 to 0.2 and from 0.2 to 0.4 each have width 0.2. They give a consistent view of the prediction scale, but their group sizes can vary substantially when predictions cluster in some ranges.
Equal-count bins are formed by sorting cases by predicted value and splitting them into groups with approximately the same number of cases. Their widths can differ: a dense region of predictions may need a narrow interval to hold the target number of cases, while a sparse region may need a wider one. This approach tends to avoid very small groups, but the plotted points no longer represent equal-length ranges of predicted values.
Ties can complicate equal-count grouping. If many cases have the same predicted value, splitting those tied cases across groups may be undesirable or impossible while keeping group sizes equal. A practical choice is to keep tied predictions together, accept unequal group sizes, and report the resulting sizes and boundaries. Whichever approach is used, the groups must be defined clearly enough that another reader can tell which cases were summarized together.
Worked Example: Three Equal-Count Bins
Consider 12 hypothetical cases with predicted event probabilities and binary observed outcomes. The predictions are already sorted. For a compact demonstration, group the cases into three consecutive bins of four.
| Case | Prediction | Outcome |
|---|---|---|
| 1 | 0.05 | 0 |
| 2 | 0.10 | 0 |
| 3 | 0.15 | 1 |
| 4 | 0.20 | 0 |
| 5 | 0.25 | 1 |
| 6 | 0.30 | 0 |
| 7 | 0.35 | 1 |
| 8 | 0.40 | 1 |
| 9 | 0.45 | 0 |
| 10 | 0.50 | 1 |
| 11 | 0.55 | 1 |
| 12 | 0.60 | 0 |
For each bin, calculate the mean prediction and mean outcome. The first bin contains cases 1–4:
Its plotted point is \((0.125,0.25)\), based on four cases. In the second bin, cases 5–8, the mean prediction is \((0.25+0.30+0.35+0.40)/4=0.325\), and the mean outcome is \((1+0+1+1)/4=0.75\). The point is \((0.325,0.75)\). In the third bin, cases 9–12, the mean prediction is \((0.45+0.50+0.55+0.60)/4=0.525\), and the mean outcome is \((0+1+1+0)/4=0.50\). The point is \((0.525,0.50)\). These are descriptive summaries of this hypothetical set, not evidence about a broader population on their own.
Changing the Number of Bins
The same data can produce a different level of detail if the number of bins changes. With more bins, the plotted summaries cover narrower prediction ranges in some parts of the scale, and local patterns may become visible. But each group contains fewer observations, so a small change in which cases fall into a bin can produce a larger change in its observed proportion. With fewer bins, each point averages over more cases, but distinct behaviors in different prediction ranges may be combined.
Worked Example: Two Broader Bins on the Same Data
Use the 12 prediction-outcome pairs above, but now make two equal-width bins: predictions from 0 through 0.30, and predictions greater than 0.30 through 0.60. The endpoint convention places a prediction of exactly 0.30 in the first bin.
The first bin contains six cases. Its mean prediction and observed event proportion are:
The second bin also contains six cases. Its mean prediction is \((0.35+0.40+0.45+0.50+0.55+0.60)/6=2.85/6=0.475\). Its mean outcome is \((1+1+0+1+1+0)/6=4/6\approx0.667\). The two points are therefore approximately \((0.175,0.333)\) and \((0.475,0.667)\).
These points are not the same as the three-bin points in the previous example because each point now summarizes a different set of cases. The broader grouping smooths over the contrast between the first four and next four cases in the three-bin display. That may make a simpler summary, but it also hides some of the variation visible with more groups. Neither display changes the underlying predictions or outcomes.
Changing Boundaries and Group Sizes
Even when the number of bins stays fixed, shifting their boundaries can move cases from one group to another and change the plotted means. This is especially consequential near a boundary: two cases with nearly identical predictions can be placed in different bins and contribute to separate summaries. A group’s size matters as well. For binary outcomes, each additional event changes the observed proportion by \(1/n_B\), so the same one-event change has a larger effect in a smaller group.
Worked Example: Shifting Three Bin Boundaries
Return to the same 12 cases and use three bins with different boundaries: predictions from 0.05 through 0.25, greater than 0.25 through 0.45, and greater than 0.45 through 0.60. The bins have sizes 5, 4, and 3, respectively.
In the first bin, the predictions sum to \(0.05+0.10+0.15+0.20+0.25=0.75\), and the outcomes sum to 2. Thus the plotted point is \((0.75/5,2/5)=(0.15,0.40)\). In the second bin, the predictions are 0.30, 0.35, 0.40, and 0.45; their sum is 1.50. The outcomes sum to \(0+1+1+0=2\), so its point is \((1.50/4,2/4)=(0.375,0.50)\).
In the last bin, the predictions sum to \(0.50+0.55+0.60=1.65\), and the outcomes sum to \(1+1+0=2\). Its point is \((1.65/3,2/3)=(0.55,0.667)\). This grouping gives three plotted points, as did the earlier equal-count grouping, but the point locations differ because the boundaries and group memberships differ. The last observed proportion is based on only three cases, so it is particularly sensitive to one outcome changing from 0 to 1 or vice versa.
Why a Bin’s Vertical Position Summarizes Error
For any bin, subtracting the mean prediction from the mean outcome gives the average observed-minus-predicted difference in that group. This follows directly from the definitions:
Thus, a point above the diagonal has a positive average observed-minus-predicted difference for that bin, while a point below the diagonal has a negative one. This calculation explains what the vertical difference represents; it does not make the bin’s summary a reliable estimate of a fine-grained pattern when the bin contains few cases.
Choosing and Reporting a Binning Scheme
A useful choice balances resolution against the amount of data supporting each point. More bins are not automatically better: many small groups can create a jagged-looking plot in which each observed proportion is based on very few outcomes. Fewer bins are not automatically better either: broad groups may conceal changes across the prediction range. The prediction distribution also matters. Equal-width bins can be nearly empty when predictions cluster, while equal-count bins can span very different prediction ranges.
Before interpreting a plot, check the actual group sizes and prediction ranges rather than relying only on the number of bins. When useful, compare a second reasonable grouping on the same cases. If a feature appears under one choice but disappears or reverses under another, describe that sensitivity instead of presenting the first display as definitive. For small samples, fewer groups may be easier to interpret, but the resulting summaries still need to be labeled as broad and limited.
A report should say how the bins were formed, give their boundaries or the rule used to determine them, and show or report the number of cases in each bin. It should also make clear how boundary values were assigned. For example, “equal-width intervals of 0.20, with a value on a shared boundary assigned to the lower interval” is more informative than simply saying “five bins.” If predictions are tied, state how the ties were handled.
Common Mistakes
- Treating one bin count as uniquely correct. A plot can look more or less smooth because of the chosen resolution. Compare plausible choices when the pattern matters.
- Ignoring group sizes. Two points based on the same number of cases need not have the same uncertainty, and a small group’s observed proportion can change substantially with one outcome.
- Leaving boundaries ambiguous. Without an endpoint rule, a case exactly on a boundary might be counted twice or not at all. Define intervals so every case is assigned once.
- Assuming equal-width bins have equal sizes. Equal widths refer to lengths on the prediction scale, not numbers of cases.
- Reading a binned point as a summary of every prediction in its range. It summarizes only the cases assigned to that bin, using their actual mean prediction and mean outcome.
Check Your Understanding
Use the definitions and examples above to answer these questions.
- For a binary outcome, what does the mean outcome in a bin represent?
- How can increasing the number of bins make a calibration plot both more informative and more variable?
- What is the difference between equal-width and equal-count bins?
- Why should a binning rule specify how predictions exactly on a shared boundary are assigned?
- If one event changes from 0 to 1 in a bin of 4 cases, by how much does that bin’s observed proportion change? How does that compare with a bin of 20 cases?