Why Examine Calibration Plots?
A binary prediction model assigns each person a predicted probability, such as a risk of disease within a specified time horizon. A prediction is calibrated when people assigned a given risk experience the outcome at about that rate. Calibration plots make this comparison visible: they place predicted risks on one axis and observed outcome frequencies on the other.
Discrimination and calibration answer different questions. Discrimination concerns whether people with the outcome tend to receive higher predictions than people without it. Calibration concerns whether the numerical probabilities correspond to observed frequencies. A model may rank people usefully while systematically predicting risks that are too high or too low. The Brier score, introduced below, summarizes squared probability errors and is affected by both aspects of performance.
Grouped Calibration Plots
A grouped plot divides validation predictions into bins, often using equal-sized groups. For each group, calculate the mean predicted probability and the observed event proportion. Plot the observed proportion vertically against the mean predicted probability horizontally. A perfectly calibrated prediction process would place points on the 45-degree diagonal.
If a point is below the diagonal, its observed event proportion is smaller than its mean predicted risk: predictions in that group are too high, or overpredicting, on average. A point above the diagonal indicates that the observed proportion exceeds the mean predicted risk: predictions are too low, or underpredicting, on average. This directional interpretation follows from comparing the vertical coordinate with the horizontal coordinate.
The grouping rule matters. Equal-sized bins give similar sample sizes per point but can cover very different risk ranges. Fixed-width bins use a common risk scale but may leave some bins sparse or empty. Report the number of people and events in each bin, since an observed proportion based on a few events is imprecise. The plot is a diagnostic, not a proof of calibration: grouping conceals variation within bins, and sampling variation can move points away from the diagonal.
Worked Example: Reading a Grouped Calibration Plot
Suppose a hypothetical validation sample contains 400 people, divided into four equal-sized groups of 100 by predicted risk. For illustration, assume everyone in each group has the group’s mean predicted risk shown below. The observed event counts are also shown.
| Group | Mean predicted risk | Events out of 100 | Observed event proportion |
|---|---|---|---|
| 1 | 0.05 | 4 | 0.04 |
| 2 | 0.15 | 13 | 0.13 |
| 3 | 0.30 | 24 | 0.24 |
| 4 | 0.42 | 44 | 0.44 |
For group 1, the observed proportion is \(4/100=0.04\), compared with a mean predicted risk of \(0.05\). Its plotted point is \((0.05,0.04)\), below the diagonal: this group shows overprediction by 0.01, or one percentage point.
Groups 2 and 3 also lie below the diagonal because \(0.13<0.15\) and \(0.24<0.30\). They indicate overprediction by 0.02 and 0.06, respectively. Group 4 lies above the diagonal because \(0.44>0.42\), indicating underprediction by 0.02. The overall pattern therefore is not a uniform shift in one direction. These are sample observations; uncertainty around each proportion would help show how precisely the points locate the underlying calibration.
Smoothed Calibration Plots
A smoothed calibration plot avoids assigning each prediction to a single bin. Instead, at each prediction value of interest, it estimates the average observed outcome among people with nearby predicted risks. The resulting curve can reveal gradual departures from the diagonal that a small number of wide bins might conceal.
One transparent smoother is a kernel-weighted local average. Choose a target predicted risk \(p_0\) and a bandwidth \(h>0\). Give person \(i\) weight \(w_i=K((\hat p_i-p_0)/h)\), where \(K\) is a nonnegative kernel function. The smoothed observed event rate at \(p_0\) is the weighted average of the binary outcomes. A triangular kernel, for example, assigns weight \(1-|u|\) when \(|u|<1\) and zero otherwise. Only predictions within distance \(h\) of \(p_0\) contribute.
Here \(\hat m(p_0)\) estimates the observed event frequency near predicted risk \(p_0\). A calibrated curve follows \(\hat m(p_0)=p_0\). Small bandwidths retain local detail but can produce a noisy curve; large bandwidths are smoother but can hide local departures. The curve near the boundaries of the prediction range may be especially uncertain because there are fewer nearby observations. A plot should state or otherwise document its smoothing method and bandwidth.
Worked Example: A Kernel-Smoothed Observed Rate
Consider six hypothetical validation records, with the predicted risk and binary outcome shown here. Estimate the smoothed event rate at \(p_0=0.20\) using a triangular kernel and bandwidth \(h=0.10\).
| Person | Predicted risk \(\hat p_i\) | Outcome \(Y_i\) |
|---|---|---|
| 1 | 0.12 | 0 |
| 2 | 0.16 | 0 |
| 3 | 0.19 | 1 |
| 4 | 0.22 | 0 |
| 5 | 0.25 | 1 |
| 6 | 0.34 | 1 |
The weight is \(w_i=\max(1-|\hat p_i-0.20|/0.10,0)\). The six weights are \(0.20,\ 0.60,\ 0.90,\ 0.80,\ 0.50,\ 0\), respectively. For example, person 3 receives weight \(1-|0.19-0.20|/0.10=0.90\), while person 6 receives zero because its prediction is more than \(0.10\) from the target.
The weighted sum of outcomes is \(0.20(0)+0.60(0)+0.90(1)+0.80(0)+0.50(1)+0(1)=1.40\). The sum of weights is \(0.20+0.60+0.90+0.80+0.50+0=3.00\). Therefore,
For this deliberately small illustration, the local smoothed event rate is about 0.467 near a predicted risk of 0.20. That difference does not establish that the model is miscalibrated: only a few records contribute, and outcomes are variable. In practice, calibration plots use much larger validation samples and should convey uncertainty where feasible.
The Brier Score
The Brier score is the mean squared difference between predicted probabilities and binary outcomes. It gives each person a contribution of \((\hat p_i-Y_i)^2\), then averages those contributions over the validation sample. A score of zero is perfect; lower scores indicate smaller squared probability errors.
Because both terms in each difference are between zero and one, the Brier score ranges from zero to one. When \(Y_i=1\), the contribution is \((1-\hat p_i)^2\); when \(Y_i=0\), it is \(\hat p_i^2\). Thus, assigning a high probability to an event that does not occur, or a low probability to an event that does occur, incurs a larger penalty than a probability closer to the observed outcome. A Brier score has no universal interpretation in isolation: its size depends in part on outcome prevalence and should be compared in a relevant validation setting.
Worked Example: Calculating the Brier Score From Grouped Data
Use the four hypothetical groups above, each of size 100, with every person in a group receiving that group’s listed predicted risk. For a group with prediction \(p\), observed event proportion \(o\), and \(n_g\) people, its mean squared error is \(o(1-p)^2+(1-o)p^2\). The first term averages errors among events; the second averages errors among non-events.
For group 1, the contribution is \(0.04(0.95)^2+0.96(0.05)^2=0.0361+0.0024=0.0385\). For group 2, it is \(0.13(0.85)^2+0.87(0.15)^2=0.093925+0.019575=0.1135\). For group 3, it is \(0.24(0.70)^2+0.76(0.30)^2=0.1176+0.0684=0.1860\). For group 4, it is \(0.44(0.58)^2+0.56(0.42)^2=0.148016+0.098784=0.2468\).
The groups are equal in size, so the overall score is the average of these four contributions:
The Brier score for these 400 hypothetical predictions is 0.1462. It is a summary of their squared probability errors, not a percentage of predictions that were correct. The score alone also does not show which risk ranges are miscalibrated; the plot provides that complementary detail.
Why the Brier Score Reflects More Than Calibration
For forecasts that take a limited set of values, the Brier score can be decomposed into reliability, resolution, and uncertainty. Suppose forecast group \(g\) has weight \(w_g=n_g/n\), common forecast probability \(p_g\), and observed event proportion \(o_g\). Let \(\bar y=\sum_g w_go_g\) be the overall event proportion. Reliability measures the weighted squared difference between observed proportions and forecasts. Resolution measures how much group event rates vary around the overall prevalence. Uncertainty depends only on that prevalence.
A smaller reliability term is better: it means forecasts are closer to observed group frequencies. A larger resolution term is beneficial because the model separates groups with different event rates; the term is subtracted. Uncertainty is set by the outcome prevalence in the evaluation sample. This decomposition makes clear why the Brier score is not a pure calibration measure: it rewards both calibration and resolution. Resolution is related to separating risk groups, but it is not identical to the area under the ROC curve.
The identity follows by expanding the groupwise squared error. For a group with forecast \(p_g\) and event fraction \(o_g\), the mean squared error is \(o_g(1-p_g)^2+(1-o_g)p_g^2=o_g-2o_gp_g+p_g^2\). The proposed decomposition has the same weighted sum, as shown below.
The cancellation uses \(\sum_gw_g=1\) and \(\sum_gw_go_g=\bar y\). If bins contain different individual predictions, treating all predictions in a bin as one common forecast generally makes this decomposition a grouped approximation rather than an exact decomposition of the individual-level Brier score. The score itself should be calculated from each person’s actual prediction and outcome.
In the worked example, \(\bar y=(0.04+0.13+0.24+0.44)/4=0.2125\). The reliability term is \(\{(0.04-0.05)^2+(0.13-0.15)^2+(0.24-0.30)^2+(0.44-0.42)^2\}/4=0.001125\). The uncertainty term is \(0.2125(1-0.2125)=0.16734375\), and resolution is \(\{(0.04-0.2125)^2+(0.13-0.2125)^2+(0.24-0.2125)^2+(0.44-0.2125)^2\}/4=0.02226875\). Hence \(0.001125+0.16734375-0.02226875=0.1462\), matching the directly calculated score.
Practical Interpretation and Common Pitfalls
Calibration plots and the Brier score are complementary. The plot shows where predicted risks depart from observed frequencies, while the score condenses individual probability errors into one number. Neither replaces the other: a single score does not locate the source of error, and a plot does not summarize all individual errors. In genetic prediction validation, describe the population and prediction horizon to which the predictions refer; calibration can differ across settings.
- Mistake: reversing the plot’s direction. Reasoning that points below the diagonal mean underprediction confuses the axes. The vertical coordinate is observed risk and the horizontal coordinate is predicted risk, so below the line means observed risk is lower: overprediction.
- Mistake: treating a grouped point as exact calibration evidence. A group proportion is subject to sampling variability, and grouping hides individual differences in predicted risk. Consider group sizes, event counts, and uncertainty before interpreting small departures.
- Mistake: calling the Brier score a calibration score. It uses both calibration and resolution, as well as prevalence-dependent uncertainty. Use a calibration plot to inspect calibration directly and interpret the Brier score as a combined probability-error measure.
- Mistake: comparing scores without context. Different outcome prevalence can change the attainable and baseline scores. State the evaluation population and compare against a suitable reference rather than treating a numerical value as universally good or poor.
Check Your Understanding
Try these questions before moving on.
- A calibration-plot point has mean predicted risk 0.30 and observed event proportion 0.24. Is it above or below the diagonal, and what direction of error does it indicate?
- Why might a small bandwidth produce a noisy smoothed calibration curve? What is the tradeoff from increasing the bandwidth?
- For a person with predicted risk 0.80 and outcome \(Y=0\), calculate that person’s Brier contribution.
- What do reliability and resolution represent in the grouped Brier decomposition, and which direction is desirable for each?
- Why is it generally not exact to calculate the Brier score using only each bin’s mean prediction and event proportion when individual predictions vary within bins?