What a Risk-Grouped Calibration Table Shows
A calibration assessment asks whether predicted event risks agree with event frequencies in a validation cohort. A risk-grouped table addresses this by dividing individuals into bins based on their predicted risk and comparing average predicted risk with observed event risk in each bin. The summary is easy to read, but it is also a summary of a summary: variation among individuals inside each bin is compressed into a few numbers.
Let \(t_0\) be a prediction horizon, and let \(\hat p_i(t_0)\) be the predicted probability that individual \(i\) experiences the event by \(t_0\). When the event status by \(t_0\) is fully known, let \(Y_i(t_0)=1\) if the event occurred by that time and \(Y_i(t_0)=0\) otherwise. For a group \(g\) containing \(n_g\) individuals, the group mean predicted risk and observed event proportion are
A grouped table displays these summaries, often alongside the group size and event count. A group is called empirically calibrated at the horizon if its observed event proportion equals its mean predicted risk. In a finite sample, differences are expected even when predictions are well calibrated in the population; conversely, a close match in a few groups does not establish that individual predictions are accurate.
For censored survival data, \(Y_i(t_0)\) may not be observed for everyone. A participant censored before \(t_0\) has unknown event status by the horizon, not a known non-event. The raw proportion among people whose status happens to be observed can therefore be biased. One must use an appropriate censoring-adjusted estimate, such as a group-level survival estimate or an inverse-probability-of-censoring weighted estimate, under the relevant censoring assumptions. The earlier material on Censoring-Adjusted Evaluation With IPCW and Choosing Censoring Models for Prediction Metrics covers those adjustments. The numerical examples below assume complete ascertainment through \(t_0\), so their observed proportions are directly calculable.
Why Grouping Can Hide Prediction Error
The central limitation follows directly from the group summaries. Define the individual residual \(e_i=Y_i(t_0)-\hat p_i(t_0)\). The group-average residual is
Thus, a group appears calibrated when its residuals sum to zero. That condition allows positive and negative residuals to cancel. It does not require each residual to be small, nor does it require predictions with similar values within the group to have similar event frequencies.
The same issue arises when groups are made wider. If two subgroups have opposite discrepancies, combining them can make the larger group look well calibrated. Grouping can also conceal heterogeneity: two people in the same bin may have very different predicted risks, even though the table reports only their average. A table can therefore look reassuring because its chosen bins smooth over the very patterns an assessment should reveal.
Worked Example: Opposing Errors Cancel in One Broad Group
Suppose four hypothetical individuals have fully observed event status by \(t_0\). Their predicted risks and outcomes are as follows.
| Individual | Predicted risk | Event by \(t_0\), \(Y_i(t_0)\) |
|---|---|---|
| A | 0.10 | 0 |
| B | 0.10 | 1 |
| C | 0.90 | 0 |
| D | 0.90 | 1 |
If all four individuals are placed in one broad group, the mean predicted risk is \( (0.10+0.10+0.90+0.90)/4=2.00/4=0.50 \). The observed event proportion is \( (0+1+0+1)/4=2/4=0.50 \). The broad group therefore matches exactly.
Now separate the two low-risk predictions from the two high-risk predictions. In the low-risk group, the mean predicted risk is \(0.10\), while the observed proportion is \(1/2=0.50\), a difference of \(0.50-0.10=0.40\). In the high-risk group, the mean prediction is \(0.90\), while the observed proportion is again \(1/2=0.50\), a difference of \(0.50-0.90=-0.40\). The two discrepancies cancel in the broad group.
The broad table's exact match does not mean the risk ranking or the individual predictions are validated. With only four observations, neither narrow group provides a precise estimate, but the example establishes the arithmetic point: aggregation can erase substantial, oppositely directed group differences.
Cut Points Change the Picture
Risk groups are created using cut points, such as equal-width risk intervals or quantiles of the predicted risks. There is no unique set of cut points implied by the data. Different choices change which individuals are averaged together, and hence change the apparent local pattern. The overall mean predicted risk and overall event proportion do not change merely because the bins change, but the group-specific comparisons do.
Worked Example: Two Binning Schemes, Different Local Summaries
Consider six hypothetical individuals with complete outcome ascertainment by \(t_0\). Their predicted risks are \(0.10, 0.20, 0.30, 0.40, 0.50, 0.60\), and their event indicators in the same order are \(0,0,0,1,1,0\).
Under Scheme A, place the first three individuals in one group and the last three in another. For the first group, the mean prediction is \( (0.10+0.20+0.30)/3=0.60/3=0.20 \), and the observed proportion is \(0/3=0\). For the second group, the mean prediction is \( (0.40+0.50+0.60)/3=1.50/3=0.50 \), and the observed proportion is \(2/3\approx0.667\).
Under Scheme B, make three groups of two adjacent risks. The first group has mean prediction \( (0.10+0.20)/2=0.15 \) and observed proportion \(0/2=0\). The middle group has mean prediction \( (0.30+0.40)/2=0.35 \) and observed proportion \(1/2=0.50\). The last group has mean prediction \( (0.50+0.60)/2=0.55 \) and observed proportion \(1/2=0.50\).
The two tables tell different local stories: Scheme A reports an observed proportion of zero in a group with mean prediction \(0.20\), while Scheme B separates the individuals and shows a middle group with an observed proportion \(0.50\) against mean prediction \(0.35\). Neither partition changes the overall observed proportion, \(2/6\approx0.333\), or the overall mean prediction, \(2.10/6=0.350\). With only six individuals, these numbers are illustrative, not a basis for concluding that either pattern is stable. The example shows why a claimed calibration pattern should not depend uncritically on one convenient partition.
Sparse Events and Unstable Group Summaries
A group with few individuals or few events can produce an observed proportion that moves sharply when a single outcome changes. A group with no events is not automatically evidence that predicted risk was zero, or that a positive prediction is indefensible. It is one possible outcome under positive event probabilities, particularly when the group's expected event count is small.
For a group with complete outcomes, its expected event count under the predictions is \(E_g=\sum_{i\in g}\hat p_i(t_0)\), while its observed event count is \(O_g=\sum_{i\in g}Y_i(t_0)\). The observed-to-expected event ratio is \(O_g/E_g\) when \(E_g>0\). It is undefined only when the expected count is zero. A defined ratio based on a sparse group can still be highly variable and should not be mistaken for a precise calibration assessment.
Worked Example: Zero Events With Positive Expected Count
Suppose five hypothetical individuals each have predicted risk \(0.10\) by \(t_0\), and no individual experiences the event by that horizon. Assume complete outcome ascertainment. The expected count is \(E_g=0.10+0.10+0.10+0.10+0.10=0.50\). The observed count is \(O_g=0\), so the observed-to-expected ratio is \(O_g/E_g=0/0.50=0\).
The ratio is defined because its denominator is positive. The group observed proportion is \(0/5=0\), compared with mean predicted risk \(0.50/5=0.10\). These summaries show a difference in this small group; they do not show that the model's risk predictions are systematically too high. If outcomes are independent and each person's true event probability equals the stated prediction, the probability of observing zero events is \(0.90^5=0.59049\). Thus zero events are quite compatible with these predictions under those assumptions. Dependence between individuals would change that probability calculation, and the small group remains too sparse for a stable conclusion.
Assumptions, Diagnostics, and Alternatives
A grouped comparison is meaningful only when the prediction and observed outcome refer to the same event definition, time origin, target population, and horizon. For direct event proportions, event status through \(t_0\) must be known for all members of the validation group. With censoring before \(t_0\), use a suitable adjusted estimate and state its assumptions. For example, an inverse-probability weighting approach requires an adequate censoring model and positivity: individuals in the relevant covariate patterns must have a nonzero probability of remaining uncensored long enough to contribute information. A group-level survival estimate also relies on censoring being independent of event time within the group, or on a method that adjusts for relevant covariates.
Useful checks include reporting each group's size, number of observed events, mean predicted risk, and the observed-risk estimation method. Examine whether conclusions change under reasonable alternative cut points. Look for groups with very few events, uneven group sizes, or a wide range of predicted risks within a bin. In censored data, also check whether censoring is concentrated in particular risk groups and whether the censoring adjustment is supported by the data. Uncertainty intervals around group estimates can help communicate imprecision, but they do not repair inappropriate grouping or a violated censoring assumption.
Risk grouping is not the only way to examine calibration. Calibration-in-the-large and calibration slopes summarize particular aspects of overall agreement and systematic over- or under-dispersion; they do not show every local departure. A smoothed calibration curve can reveal patterns across the risk range without imposing a few fixed cut points, though its appearance depends on the smoothing method and it can be unstable where data are sparse. No display eliminates sampling uncertainty. Grouped tables are useful as transparent descriptive summaries, especially when counts are shown, but should be read alongside other calibration diagnostics rather than treated as a definitive test.
The practical value of grouping is communication: readers can see simple observed-versus-predicted summaries. Its limitation is the same simplicity. A table's apparent clarity can encourage more certainty than the data warrant, especially when cut points were selected after looking at results or when events are scarce. Report the grouping rule, avoid treating one arbitrary partition as conclusive, and interpret each cell in light of its size and outcome ascertainment.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A group has four predicted risks of \(0.20\), and one observed event by the horizon. What are the mean predicted risk and observed proportion?
- Why can a group with equal mean predicted and observed risks still contain inaccurate predictions?
- A group has zero observed events and expected count \(0.5\). Is its observed-to-expected ratio undefined?
- What should change in the calculation if some individuals are censored before the prediction horizon?
- Why is it useful to repeat a grouped calibration assessment using a reasonable alternative set of cut points?
Solutions. (1) The mean predicted risk is \( (0.20+0.20+0.20+0.20)/4=0.20 \); the observed proportion is \(1/4=0.25\). (2) Individual residuals can have opposite signs and average to zero, so group agreement does not imply individual agreement. (3) No. The ratio is \(0/0.5=0\), which is defined; it remains a sparse and potentially unstable summary. (4) Do not code a person censored before the horizon as a non-event. Use a suitable censoring-adjusted estimate under its assumptions. (5) If local conclusions change substantially, the apparent pattern may depend on arbitrary aggregation rather than a robust feature of the predictions.