Calibration Checks for Predicted State-Occupation Probabilities

What a Calibration Check Compares

A predicted probability can look plausible without agreeing with what happens in the target population. Calibration checks this agreement: among people assigned similar predicted probabilities of occupying a particular state at a specified horizon, is the observed frequency of that state similar? The answer must be tied to a clearly specified landmark cohort, prediction horizon, and state. A model can be well calibrated for one landmark cohort and poorly calibrated for another.

Suppose people who are in a defined eligible population at landmark time \(s\) receive predicted probabilities for their state at horizon \(s+u\). Let \(K\) denote the number of possible states, and index the states by \(k=1,\ldots,K\). For person \(i\), let \(Y_{ik}(u)\) equal 1 if that person occupies state \(k\) at \(s+u\), and 0 otherwise. Write the predicted probability as \(\hat{p}_{ik}(u)\). At the horizon, the state indicators are mutually exclusive and exhaustive, so \(\sum_{k=1}^{K}Y_{ik}(u)=1\); valid predicted probabilities likewise satisfy \(\sum_{k=1}^{K}\hat{p}_{ik}(u)=1\).

Definition: State-specific calibration at a landmark and horizon means that, among people with a given predicted probability vector, the expected indicator for state \(k\) equals the predicted probability for that state. Formally, \(E\{Y_{ik}(u)\mid \hat{\boldsymbol{p}}_i(u)\}=\hat{p}_{ik}(u)\), for each state \(k\). A practical calibration check compares predicted probabilities with observed state frequencies in the target landmark cohort, overall and across groups with differing predictions.

This definition is a population property; a finite validation sample can only provide evidence about it. A useful first summary is calibration-in-the-large, which compares the cohort’s average predicted probability for a state with its observed frequency. Agreement in this summary is necessary for overall calibration, but it is not enough: overprediction for one subgroup can cancel underprediction for another. Calibration is also distinct from discrimination, which concerns whether predictions rank people with different outcomes appropriately.

Conditions for a Meaningful Check

Before comparing numbers, define who belongs to the cohort, which prediction time and horizon are being assessed, and what counts as occupying each state. The state definition must cover all possible outcomes at the horizon, with no person counted in two states. Predictions must have been generated using information available at the landmark, not information learned afterward. The validation observations should represent the population to which the predictions are intended to apply.

Ideally, predictions are assessed in data not used to fit or select the model. If the same data were used for fitting and checking, apparent agreement may partly reflect overfitting. A held-out validation sample, an external validation cohort, or a suitable resampling procedure can address this concern; uncertainty should reflect model fitting when that source of uncertainty matters.

If each cohort member’s state at the horizon is observed, the observed frequency of state \(k\) is the number in that state divided by the cohort size. If some members are censored before the horizon, simply discarding them is generally not justified: their observed outcomes may not represent the whole cohort. A censoring-adjusted estimate requires assumptions and a method suited to the censoring process. One option, detailed below, uses inverse-probability-of-censoring weights.

For that method, assume censoring is independent of the state at the horizon conditional on measured predictors of censoring and outcome, and that each relevant person has a positive probability of remaining observed through the horizon. If these conditions are not credible, a weighted frequency does not solve the problem. Diagnostics should include examining follow-up by relevant covariates and checking that estimated probabilities of remaining observed are not near zero. Large weights can make an estimate unstable.

Overall and Grouped Comparisons

With complete horizon outcomes for \(n\) people in one landmark cohort, define the observed frequency for state \(k\) as \(\bar{Y}_k(u)=n^{-1}\sum_iY_{ik}(u)\), and the average predicted probability as \(\bar{p}_k(u)=n^{-1}\sum_i\hat{p}_{ik}(u)\). The difference \(\bar{Y}_k(u)-\bar{p}_k(u)\) is an easy-to-interpret calibration-in-the-large discrepancy: a positive value means the observed frequency exceeds the average prediction. Report it in percentage points, alongside the two quantities being compared.

\bar{Y}_k(u)=\frac{1}{n}\sum_{i=1}^{n}Y_{ik}(u), \qquad \bar{p}_k(u)=\frac{1}{n}\sum_{i=1}^{n}\hat{p}_{ik}(u), \qquad \text{discrepancy}_k=\bar{Y}_k(u)-\bar{p}_k(u).

A grouped check looks for discrepancies hidden by these overall averages. For example, sort people by their predicted probability for state \(k\), divide them into groups, and compare each group’s mean prediction with its observed frequency. The comparison is state-specific: in a group defined by predicted probability of state \(k\), the observed frequency is the proportion actually in state \(k\). Plotting observed frequencies against mean predictions gives a calibration plot; perfect agreement lies on the 45-degree line. Grouping is a summary, not a test of the full conditional calibration definition, and results can depend on the chosen groups.

Each state should be checked within each landmark cohort of interest. Pooling landmarks can mask problems if the cohorts differ in their state distributions or predictions. Also check that the state-specific results are coherent: across states, observed frequencies and average predicted probabilities each sum to one. A discrepancy in one state must be balanced by discrepancies in other states when all categories are included.

Why the Overall Comparison Can Miss Miscalibration

The grouped comparison follows directly from averaging. If a cohort is divided into groups \(g\), its overall observed frequency is the group-size-weighted average of the group frequencies; its mean prediction is the same weighted average of the group mean predictions. Consequently, positive and negative group discrepancies can cancel in the overall comparison.

\bar{Y}_k(u)-\bar{p}_k(u) = \sum_g \frac{n_g}{n} \left\{\bar{Y}_{kg}(u)-\bar{p}_{kg}(u)\right\},

where \(n_g\) is the number of people in group \(g\), \(\bar{Y}_{kg}(u)\) is the observed frequency for state \(k\) in that group, and \(\bar{p}_{kg}(u)\) is the group’s mean predicted probability for that state. This identity explains why calibration-in-the-large is informative but cannot establish calibration across the range of predictions.

Worked Example: Calibration-in-the-Large in a Complete Cohort

Consider a hypothetical validation cohort of 20 people at one landmark, with all states known at a chosen horizon. There are three possible states. The observed counts are 12, 5, and 3, respectively. The average predicted probabilities for the same states are 0.55, 0.25, and 0.20.

The observed frequencies are \(12/20=0.60\), \(5/20=0.25\), and \(3/20=0.15\). Subtracting the average predictions gives discrepancies of \(0.60-0.55=0.05\), \(0.25-0.25=0\), and \(0.15-0.20=-0.05\). Thus the observed frequency exceeds the prediction by 5 percentage points for state 1, agrees for state 2, and is 5 percentage points below the prediction for state 3. The discrepancies sum to zero, as they must when the states are exhaustive and both sets of probabilities sum to one.

These results describe overall agreement only. They do not show whether predictions are accurate at both low and high risk levels. For example, overprediction among some people and underprediction among others could produce the same average result.

Worked Example: Grouped Checks Reveal Opposing Errors

In a second hypothetical cohort of 20 people, divide participants into two groups of 10 according to their predicted probabilities. For three states, the low-prediction group has mean predictions \((0.70,0.20,0.10)\) and observed counts \((6,3,1)\). The high-prediction group has mean predictions \((0.20,0.60,0.20)\) and observed counts \((3,5,2)\). Each group’s predictions sum to one, and each group’s observed counts sum to 10.

For the low-prediction group, observed frequencies are \((0.60,0.30,0.10)\). The observed-minus-predicted differences are \((-0.10,0.10,0)\). For the high-prediction group, frequencies are \((0.30,0.50,0.20)\), giving differences \((0.10,-0.10,0)\). These opposing differences show why the groups should not be replaced by a single cohort average.

Because the groups are equal in size, the cohort’s average predictions are \((0.45,0.40,0.15)\), and its observed frequencies are also \((0.45,0.40,0.15)\). Overall, calibration-in-the-large is exact for all three states, yet the grouped comparisons show deviations of 10 percentage points in opposite directions for states 1 and 2. A calibration plot or grouped table would make that pattern visible.

Adjusting Observed Frequencies for Censoring

When outcomes are not known for everyone at the horizon, an inverse-probability-of-censoring weighted estimate can reweight people whose state is observed. Let \(R_i(u)=1\) if person \(i\)'s state at the horizon is observed and 0 otherwise. Let \(Z_i\) denote measured variables used to model censoring, and let \(G_i(u)=P\{R_i(u)=1\mid Z_i\}\). A normalized weighted frequency for state \(k\) is

\hat{q}_k(u)= \frac{\sum_{i=1}^{n} R_i(u)Y_{ik}(u)/\hat{G}_i(u)} {\sum_{i=1}^{n} R_i(u)/\hat{G}_i(u)}.

Here \(\hat{G}_i(u)\) is an estimate of the probability that person \(i\)'s state is observed through the horizon. The normalized form makes the estimated frequencies sum to one when each observed person belongs to exactly one state. Compare \(\hat{q}_k(u)\) with the average predicted probability in the full target cohort. This comparison is justified only when the censoring assumptions are credible and the censoring model is adequate.

The reason for weighting can be seen from a simple conditional expectation. If censoring is independent of \(Y_{ik}(u)\) given \(Z_i\), and \(G_i(u)\) is correctly specified, then

E\left\{\left.\frac{R_i(u)Y_{ik}(u)}{G_i(u)}\right|Y_{ik}(u),Z_i\right\} = \frac{Y_{ik}(u)E\{R_i(u)\mid Z_i\}}{G_i(u)} = Y_{ik}(u).

Thus the weighted state indicator has the correct expectation under those conditions. Normalizing by the sum of weights yields a ratio estimator that is consistent under standard regularity conditions. In practice, estimated weights add uncertainty, and very small \(\hat{G}_i(u)\) values can cause a few people to dominate the calculation. Report how censoring was handled and assess weight stability rather than presenting the weighted result as assumption-free.

Worked Example: A Censoring-Adjusted Frequency

Suppose a hypothetical landmark cohort contains five people, but only four have their state observed at the horizon. The estimated probabilities of remaining observed for these four people are \(0.80, 0.80, 0.50,\) and \(0.50\). Their observed states are, in order, state A, state A, state B, and state D. The predicted probabilities averaged across all five cohort members are \(0.46\), \(0.32\), and \(0.22\) for A, B, and D.

The inverse-probability weights are \(1/0.80=1.25\), \(1.25\), \(1/0.50=2\), and \(2\). The weighted state totals are \(1.25+1.25=2.50\) for A, \(2\) for B, and \(2\) for D; the total weight is \(1.25+1.25+2+2=6.50\). Therefore the adjusted frequencies are \(2.50/6.50=0.385\) for A, \(2/6.50=0.308\) for B, and \(2/6.50=0.308\) for D. They sum to 1.001 because of rounding.

Using the unrounded fractions, observed-minus-predicted differences are \(0.385-0.46=-0.075\) for A, \(0.308-0.32=-0.012\) for B, and \(0.308-0.22=0.088\) for D. These estimates suggest lower observed frequency than predicted for A and B, and higher observed frequency for D. They are not definitive evidence of miscalibration: with only four observed outcomes, uncertainty is substantial, and the censoring assumptions and estimated weights matter.

Uncertainty, Diagnostics, and Alternatives

A frequency from a small group can vary considerably even when the predictions are well calibrated. Show group sizes alongside grouped results, and use uncertainty intervals when appropriate. A simple binomial interval treats the group’s outcomes as independent Bernoulli observations for the state being checked; it does not account for fitting the prediction model, estimating censoring weights, or dependence from repeated observations on the same person. A bootstrap that repeats the relevant fitting and weighting steps can reflect more of these sources of uncertainty when its resampling design matches the data structure.

Useful diagnostics include checking the number of people and observed outcomes in each group, examining whether grouped results change under reasonable alternative groupings, and looking for influential censoring weights. Sparse states or horizons with little follow-up may make calibration difficult to assess. A smooth calibration curve can supplement grouped summaries, but it is not a remedy for sparse data and depends on the smoothing method.

The main alternatives answer different questions. Calibration-in-the-large is compact but can conceal subgroup errors. Grouped tables and plots show where discrepancies occur, but depend on grouping and may be noisy. A formal goodness-of-fit test can summarize evidence against a specified calibration null, but its result may depend on binning, and a large p-value does not establish good calibration. These approaches do not replace examination of validation design, state definitions, or follow-up.

For a given cohort, assess each state at the chosen horizon; repeat the check separately for other horizons or landmark cohorts that matter to the intended use. These are point-in-time checks, not proof of calibration at every time or in every subgroup. Calibration may also change when the target population changes. If the same people contribute to multiple landmark cohorts, results across cohorts are dependent and should not be interpreted as independent replications.

Common Pitfalls

Relying only on overall agreement. Equal cohort averages can conceal offsetting errors across prediction groups. Examine grouped comparisons as well.
Dropping people censored before the horizon. The observed subset may be systematically different. Use a justified censoring adjustment or restrict the claim to a setting with complete outcomes.
Pooling landmarks or states without explanation. Cohort-specific or state-specific calibration problems can disappear in pooled summaries. Define and report the target for every comparison.
Treating a wide interval or non-significant test as proof of calibration. These may indicate limited information. Report the estimates and their uncertainty, and describe the range of predictions and group sizes.
Using validation data that helped fit the model without accounting for optimism. Apparent agreement in development data can be overly favorable. Prefer independent validation or a suitable procedure that accounts for model fitting.

Key takeaway: For each state, landmark cohort, and horizon, compare the mean predicted probability with an observed frequency, then examine whether that agreement holds across prediction levels. Incomplete follow-up, sparse data, and model-fitting uncertainty can all limit what the check supports.

Check Your Understanding

Use the definitions and calculations above to answer these questions. Brief solutions follow the questions.

For one state, a cohort has an observed frequency of 0.42 and an average predicted probability of 0.37. What is the calibration-in-the-large discrepancy, and how should it be interpreted?
Why can a cohort have matching overall observed and predicted frequencies even when its grouped calibration checks show discrepancies?
In a complete group of 25 people, 8 occupy a specified state at the horizon. What is the observed frequency for that state?
Name two assumptions needed for the censoring-weighted frequency and explain why very small probabilities of remaining observed are concerning.

Solutions. (1) The discrepancy is \(0.42-0.37=0.05\), or 5 percentage points; the observed frequency exceeds the average prediction. (2) Positive and negative group discrepancies can cancel in the size-weighted overall average. (3) The observed frequency is \(8/25=0.32\), or 32%. (4) Examples are conditional independence of censoring and the horizon state given measured predictors, and positivity of remaining observed through the horizon. Small probabilities create large inverse weights, making the estimate unstable and highly influenced by a few observations.