Calibration Of Transition Probabilities

What Calibration Means for a Transition

A transition probability is useful only if it agrees with what happens in the population to which it is applied. For a specified origin state, destination state, and prediction horizon, calibration asks whether subjects assigned a probability near a given value experience that transition at about the predicted frequency. This is an assessment of agreement, not a measure of whether the model ranks subjects well.

Fix an origin state \(a\), a destination state \(b\), and a horizon \(\tau\). Consider subjects who are eligible for prediction from state \(a\) at the same prediction origin. For subject \(k\), let \(r_k\) be the model’s predicted probability of making the specified transition from \(a\) to \(b\) by \(\tau\), under the transition-probability target defined for that setting. Define \(Y_k(\tau)=1\) if that transition has occurred by \(\tau\), and \(Y_k(\tau)=0\) otherwise. A competing transition before the specified transition makes \(Y_k(\tau)=0\) for this target; it must not be counted as the specified transition.

For group \(g\), let \(G_g\) be its set of subjects and \(n_g\) its size. The mean predicted probability, observed transition frequency, and calibration difference are

A positive \(\hat D_g\) means the observed frequency exceeds the group’s average prediction; a negative value means it falls below it. Plotting \(\hat O_g\) against \(\bar r_g\) gives a grouped calibration display: perfect grouped agreement lies on the 45-degree line. Grouping is an approximation. Even if each group agrees on average, miscalibration within a group can be hidden.

Assumptions and What Counts as an Observed Outcome

Before calculating frequencies, the prediction target and outcome ascertainment must match. Every subject in a comparison should have the same origin-state definition, destination transition, and horizon. Predictions must refer to that same target. If a subject’s transition status through \(\tau\) is unknown because follow-up ended earlier, that subject cannot simply be coded \(Y_k(\tau)=0\). A simple observed frequency is justified when statuses through the horizon are known for all included subjects. With incomplete follow-up, use an appropriate method for estimating observed transition probabilities that accounts for competing outgoing transitions and censoring; the raw proportion among subjects with known status can be biased.

For a direct uncertainty calculation, independent subjects are not by themselves enough to justify a binomial standard error. If outcomes in a group are independent and identically distributed Bernoulli variables with a common event probability \(p\), then the group count is binomial and the usual estimated standard error of \(\hat O_g\) is \(\sqrt{\hat O_g(1-\hat O_g)/n_g}\). If subjects instead have different event probabilities, the count is generally Poisson-binomial, not binomial. Independence still permits a variance calculation, but it depends on the individual probabilities.

Why the Standard Error Depends on the Outcome Model

Let \(p_k=P\{Y_k(\tau)=1\}\) for each subject in a group. If the outcomes are independent Bernoulli variables, then \(E\{Y_k(\tau)\}=p_k\) and \(\operatorname{Var}\{Y_k(\tau)\}=p_k(1-p_k)\). Since the variance of a sum of independent variables is the sum of their variances, dividing the sum by \(n_g^2\) gives

Only when all \(p_k\) are equal to a common \(p\) does this reduce to \(p(1-p)/n_g\), the binomial variance. In real validation data, the individual true probabilities are not known. Do not substitute the pooled observed frequency into the homogeneous binomial formula merely because subjects are independent. Depending on the design, uncertainty can instead be assessed using a suitable model-based variance estimate or resampling procedure that preserves the relevant sampling structure. If subjects are clustered, resampling or variance estimation must also account for that dependence.

Reading a Grouped Calibration Assessment

The following hypothetical example uses three prediction groups. All subjects share a prediction origin, transition target, and 12-month horizon; follow-up is complete through 12 months. Competing transitions are recorded as outcomes other than the specified transition, and the table reports only whether the specified transition occurred. The table is a descriptive comparison; no standard errors are inferred from independence alone.

To construct groups, sort predictions and choose a grouping rule before examining outcomes where feasible. Groups with roughly equal numbers of subjects can stabilize group sizes, while fixed probability ranges make boundaries easier to compare across samples. Either choice trades detail against precision. A few large groups can conceal local departures; many small groups produce noisy frequencies and can make the display look irregular by chance.

Diagnostics, Alternatives, and Limitations

A grouped display is most informative when interpreted alongside checks that the assessment was set up correctly. Verify that subjects really share the stated prediction origin and horizon, that the target transition is distinguished from competing transitions, and that outcomes are known or estimated using an appropriate procedure. Check group sizes and the range of predictions within groups. A wide range within a group makes its single mean prediction a crude summary. Repeat the descriptive assessment using a small number of reasonable grouping rules to see whether the broad pattern depends on arbitrary cut points; do not select a rule solely because it gives the preferred result.

Calibration answers a different question from several other validation summaries. A model can order subjects well but systematically overpredict, or have acceptable group calibration while ranking subjects poorly. Calibration-in-the-large summarizes overall agreement rather than how agreement changes across prediction levels; a smoothed calibration curve avoids fixed bins but requires its own smoothing choices and can be unstable where data are sparse. The Brier score summarizes squared prediction error and combines aspects of calibration and separation rather than displaying the groupwise frequency relationship. These summaries complement, rather than replace, the transition-specific grouped comparison.

The main limitations are finite-sample noise, dependence on the grouping rule, and inability to detect within-group miscalibration. A calibration display is also specific to the transition, origin, horizon, and population evaluated. It does not establish that the same predictions are calibrated at another horizon or in a different population. With censoring or other incomplete outcome ascertainment, naive complete-case frequencies can be misleading; the outcome-estimation method and its assumptions must be reported. Finally, agreement does not show that predictions caused the outcomes, nor does a small observed difference prove that meaningful miscalibration is absent.